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Welcome 


WHAT ABOUT THE CATS? 


Dear Reader, 


The sorry state of discourse on the Internet has many causes, 
and the experts have offered many solutions for the mess. 
One of the biggest problems is the way that social media 
platforms reward engagement. You get shown different con- 
tent depending on what you click on. For instance, you might 
click on an innocuous-looking link for a topic such as “vac- 
cine side effects” and from there be shown slightly darker 
stuff until you descend down into a thought bubble that rep- 
resents a different reality from what other users are seeing. 


| know what you are thinking: Is it possible for the platform 
(and the user) to get better at spotting these innocuous links 
that act as a front door to the dark conspiracies that have en- 
trapped so many users? My best response is “good luck with 
that one,” or, to put it another way, what are you going to do 
about the cats? 


Those adorable cat videos have been a fixture of the World 
Wide Web for years. One good video of a kitten playing a 
piano or scaring off a bear can generate thousands or even 
millions of clicks. If you know that, and | know that, it should 
not be a surprise that the people who drive traffic to conspir- 
acies and other sketchy content are also aware of the power 
of cute. 


Internet watchers are becoming increasingly alarmed by the 
connection between fringe groups and cute animal videos. A 
recent article in The New York Times [1] offers a useful sum- 
mary of some recent examples. (Many New York Times sto- 
ries are paywalled, but this one is accessible — at least for 
now.) According to analysis by the Times, Epoch Media, an 
organization associated with the Falun Gong religious sect 
that has expressed support for OQAnon and other conspiracy 
theories, has posted 12,062 cute animal videos on 103 of its 
Facebook pages over the past year. These posts have racked 
up nearly 4 billion views. The website of a prominent anti- 
vaxxer, who researchers named as one of the most prolific 
sources of COVID-19 misinformation, recently ran a story 
with the title “Kitten and Chick Nap So Sweetly Together.” 


Info 


[1] “Those Cute Cats Online? They Help Spread Misinformation” 
by Davey Alba, The New York Times, December 2021, 
https:/www.nytimes.com/2021/12/0 1/technology/ 
misinformation-cute-cats-online.html 


The use of cute animal videos as engagement bait poses a 
problem for all who are working to rein in the abuses of In- 
ternet misinformation. These videos are endlessly popular, 
and nothing about the content gives any indication of the 
owner's motivations. Sometimes these videos and cute im- 
ages are accompanied by seemingly innocent teasers that 
say, “If you like videos like this one, click here,” and suddenly 
the viewer subscribes to a service or channel that opens a 
conduit for more misinformation. 


Facebook and Google think they will eventually solve prob- 
lems like this using Al, but such solutions are currently a long 
way off. What can we do about it now? It might be worth re- 
membering to check the source before you forward that cute 
puppy pic to all your relatives and friends. 


he 


Joe Casad, 
Editor in Chief 
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Explore machine learning 
with a simple but illuminating 
missing person app. 
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Skydive 

Use this free traffic analyzer 
to visualize your network 
and track down problems. 


News 

e EndeavorOS 21.4 Has Arrived 

e NixOS 21.11 Now Available for Download 

¢ KDE Plasma Developers Introduce a Gnome-Like Overview 
e Rocky Linux 8.5 with Secure Boot Support 

¢ CronRAT Malware Targets Linux Servers 

e AlmaLinux OS 8.5 Now Available 


Kernel News 
e Supporting New Hardware Features: UINTR 
¢ Grooming Corporate Filesystem Maintainers 


Fuzz Testing 

Fuzzing is an important method for finding bugs and 
security vulnerabilities in software. Read on to find out 
what fuzzing is and which methods are commonly used 
today. 


Distro Walk — Parrot OS 

Parrot OS offers a more secure desktop with practical tools 
for both newbies and veteran users that encourage better 
security. 
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A free and freewheeling 
unofficial community 
repository for Ubuntu users. 


PiMiga 1.5 
Commodore Amiga lives on 
with this playful emulator 
for the Raspberry Pi. 
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Access the trove of geodata 
available online to create 
your own maps and 
navigational documents. 
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Command Line — PDF Security 

PDFs, the preferred format for file sharing, only offer 
primitive privacy and security. With these command-line 
tools, you can help your PDFs meet modern security 
requirements. 


Charly’s Column — getnews.tech 

Instead of websites or newsfeeds, Charly prefers to use 
getnews.tech at the command line to keep up to date with 
what's happening around the world. 


Programming Snapshot — Game Development 
We all know that the Fyne framework tor Go can be used to 
create GUls for the desktop, but you can also write games 
with It. Mike Schilli takes on a classic from the soccer field. 


Machine Learning 
We explore some important machine learning techniques 
with a simple missing person app. 


Skydive 

If you don't speak fluent Ethernet, it sometimes helps to 
get a graphical view of what your network is doing. 
Skydive offers visual insights that could reveal complex 
error patterns. 


Pacstall 

Many users wish Ubuntu had a free and easily accessible 
user-driven package repository like Arch’'s AUR. Pacstall 
steps into the gap. 
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Ever wonder how attackers 
discover those “carefully 

crafted input strings” that crash 
programs and surrender control? 
Welcome to the world of fuzz 
testing. We introduce you to the 
art of fuzzing and explore some 


leading fuzz testing techniques. 
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56 Elixer 
The Elixir programming language on a Raspberry Pi lets you 
create distributed projects in just a few lines of code. 


PiMiga 1.5 
Convert a Raspberry Pi 400 into a retro computer that 
behaves like the popular Amiga 500. 
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69 Welcome 


This month in Linux Voice. 


71 Doghouse — New/Old Computers 
Computer architectures from the 1960s and ‘70s are 
given new life via modern kits. 


72 Slideshows with Kdenlive 
Kdenlive plays to its strengths when editing larger 
video projects and also helps users create appealing 
slideshows with impressive effects. 


78 Filesharing to Go 
If you want to exchange Tiles over the local network, 
you do not have to set up a file server. A number of 
handy tools let you drag and drop to send files. 


FOSSPicks 
This month Graham looks at Plasma System Monitor, 
projectM audio visualizer, yt-dlg downloader GUI, and more. 


Tutorial — prettymaps 
Prettymaps combines multiple Python libraries to 
make it easy to draw maps straight from the 
OpenStreetMap database. 
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Parrot OS 4.11 
64-bit 
The Parrot Project is an Italian-based distribution based 
on the Debian Testing repository with an emphasis on 
security. Although regular releases are made, Parrot OS 
is also a rolling release, enhancing security with the 
latest updates. 


Parrot OS is available in three editions: Security, Home, 
and ARM. The Security Edition, which ts set up for 
everything from penetration testing to digital forensics 
and reverse engineering, includes a wide variety of 
programming tools. The Home Edition on this month's 
DVD includes many of the security features found on the 
Security Edition. The Home Edition features Tor (aka The 
Onion Router), the well-known tool for anonymous web 
browsing; OnionShare, which shares files of any size 
over Tor; and AnonSurfer, which routes the operating 
system's communications over Tor. Other tools include 
the Electrum Bitcoin Wallet for secure trading of crypto 
currencies and the self-explanatory metadata cleaner 
and secure file deleter. 


Parrot OS deals with many advanced security concepts 
and tools. However, by placing the tools on a desktop 
environment, Parrot makes privacy and security 
accessible to users of all levels. If you are a new user, 
Parrot is one of the best distributions to teach you what 
you need to know to keep your computing safe. 


Defective discs will be replaced. 
Please send an email to subs@linux-magazine.com. 
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Fedora Workstation 35 
64-bit 
Fedora, one of the best-known Linux distributions, 
functions as the testing ground for Red Hat Enterprise 
Linux. Fedora is known for introducing new features 
that often become standard on other distributions, 
and Fedora Workstation 35 is no exception. 


Many of the new features are due to the introduction of 
Gnome 41. Not only does Gnome 41 include additional 
support for Wayland, but it also offers a revamped app 
for installing software; the Connections app, which con- 
figures remote connections over VNC and RDP; Gnome'’s 
Mobile Network panel for cellular connections; and 
Parental Controls. Another new feature is a Multitasking 
panel for configuring common settings for switching 
apps and workspaces. By default, power modes are 
enabled to give users the choice of increasing their 
machine’s environmental friendliness at a minor cost 
in performance. Outside of Gnome, Fedora Workstation 
includes several new games; enhanced Bluetooth sup- 
port for PipeWire, the audio and video server introduced 
in the previous release; and increased Wayland support 
for NVidia graphics cards. In addition, a repository for 
Flatpak packages is now enabled by default, along with 
the traditional repositories for RPM packages. 


If you have used Fedora, you know that you cannot go 
wrong installing it on your desktop. If you are new to 
the distro, you are in for a pleasant surprise, no matter 
what your level of expertise. 


Although this Linux Magazine disc has been tested and is to the best of our knowledge free of malicious software and defects, Linux Magazine 
cannot be held responsible and is not liable for any disruption, loss, or damage to data and computer systems related to the use of this disc. 
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Windows PC, a Mac, or a 
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Arrived 


e NixOS 21.11 Now 
Available for Download 


) e¢ KDE Plasma Developers 


Introduce a Gnome-Like 
Overview 

e Rocky Linux 8.5 Now 
Available with Secure 
Boot Support 

e More Online 


(\ © CronRAT Malware Targets 


Linux Servers 


e AlmaLinux OS 8.5 Now 
Available 


§ EndeavorOS 21.4 Has Arrived 


If you prefer your Linux to be of the Arch-type, but don’t want to go through the 
challenges inherent in installing the full-blown Arch Linux, you have options. One 
such option is EndeavorOS. Endeavor OS calls itself “terminal-centric.” That 
doesn't mean you'll be spending all of your time within the terminal. In fact, I'd 
say that Endeavor OS labeling itself as such is a bit misleading. I’ve worked with 
the OS and found it quite easy to use. 

But what does the new “Atlantis” version of EndeavorOS have to offer? First and 
foremost, it ships with kernel 5.15, which is bleeding edge. One very important feature 
found in this kernel is the newly written NTFS3 
driver, which vastly improves how Linux can inter- 
act with NTFS filesystems. 

Other improvements include a new sanity 
check for NVidia and kernel updates, improve- 
ments to the Welcome app, and the ability to 
easily delete the cache of uninstalled packages. 
In addition, the Calamares installer can read out- 
put from pacman actions, randomize EFI path 
naming, install Xfce and i3 at the same time, en- 
able NVidia DRM mode setting by default. Also, 
Btrfs now uses zstd for installation on both 
SSDs and HDDs, PipeWire is now enabled by default, and Legacy/BIOS boot uses 
a fixed label name so it’s now compatible with older BIOS systems. 

To find out more about the Atlantis release of Endeavor OS, read through the 
release notes (httos://endeavouros.com/news/the-atlantis-release-is-in-orbit/) and 
download an ISO for installation (httos:/endeavouros.com/latest-release/). 





ENDEAVOURYS 


BNixOS 21.1] Now Available for Download 


NixOS is a bit different than most Linux distributions, because of a unique approach 
to package and configuration management. NixOS uses the Nix package manager to 
build everything — even the kernel. And even the entire system configuration (from 
fstab, users, services, firewalls, and more) Is taken care of from within a single, 
global configuration file. This one-two punch makes NixOS very complex. In fact, 
many consider it on the same level as Gentoo. 

In other words, NixOS is not for the faint of heart. 

With NixOS 21.11, there are plenty of new features to be found, including kernel 
5.10, nf_tables (in place of iptables), KDE Plasma on Wayland, PHP 8.0, Python 3.9, 
PostgreSOL 13, Spark 3, Bash 5.0, Gnome 41, systemd 249, Pantheon 6, Kuber- 
netes Helm 3.7.0, OpenSSH 8.8p1, and general LXD improvement. 

The one piece of the puzzle that was not upgraded was Nix. Because of regres- 
sions in non-experimental behaviors, NixOS ships with Nix 2.3.16 (instead of the 
latest version, 2.4). 
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Download your copy of NixOS from the official download page (https://nixos.org/ 
download. html) and read the release notes (httos:/nixos.org/manual/nixos/stable/ 
release-notes.html#sec-release-21.11) for more information. 


CC Plasma Developers Introduce a 
Gnome-Like Overview 


KDE Plasma is a very user-friendly desktop interface found on several popular Linux 
distributions. But there are a couple of features that have needed some slight 
tweaking for a while. Those features are how users interact with both activities and 
virtual desktops. 

With the release of KDE Plasma 5.24, that all changes as both features are getting 
a rather Gnome-like makeover. A new overview screen will offer a full-screen view 
of both virtual desktops and all currently open applications. Along with these two 
features, a search bar will be included with the 
overview that allows users to find applications, 
tiles, browser tabs, documents, and more. 

Although this might require longtime users to 
have to adjust their workflows, if the imple- 
mentation is as successful as what Gnome did 
with version 40, most will be quite happy with 
how efficient it is. 

KDE Plasma 5.24 Is set to release February 
3, 2022, and will include other new features, 
such as custom highlighting, DRM lease, fin- 
gerprint reader support, support for the NVidia 
driver's GBM back end, new screenshot fea- 
tures in Spectacle, and a global keyboard shortcut to move a window to the cen- 
ter of the screen, as well as plenty of bug fixes and performance improvements. 


Bait Linux 8.5 Now Available with Secure 
Boot Support 


Soon after Red Hat Enterprise Linux 8.5 was released, AlmaLinux 8.5 Stable was 
made available. Not to be outshined, the original developer of CentOS has un- 
leashed the 8.5 version of Rocky Linux, which introduces a crucial feature for 
mass adoptions, Secure Boot support. 

Main developer (and original creator of CentOS), Gregory Kurtzer, says of this re- 
lease, “There was an amazing amount of work and collaboration that went into this 
release. The Rocky Release Engineering team went far and above the call of duty to 
make 8.5 a reality so quickly. ” 

The one caveat to the Secure Boot option is that it must be activated after installa- 
tion. For this, admins will need to run the commands: 


sudo dnf install -y keyutils 
sudo Keyct! show %:.platform 
sudo mokutil --sb 


Other additions to Rocky Linux 8.5 include the FastestMirror DNF plugin (for 
faster network installations), Thunderbird with PGP support, Raspberry Pi aarch64 
support, an enhanced Cockpit web console, better container support, new system 
roles, OpenJDK 17 support, and the newly-added Network Time Security protocol 
for use with NTP. 


ON Rocky Linux" 


Download a copy of 
Rocky Linux 8.5 from the 
official website and read 
through the entire release 
notes for more information. 
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File Compression for HPC 

e Jeff Layton 

One of the most underutilized capabilities 
in Linux and HPC is compressing unused 
data files. 
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Multicloud Management with Ansible 

e Andreas Stolzenberger 

Remain independent of your cloud provider 
by automatically rolling out virtual machines 
and applications with Ansible neutral 
inventory files. 


Zero-Ops Kubernetes with MicroK8s 

¢ Chris Binnie 

A zero-ops installation of Kubernetes with 
MicroK8s operates on almost no compute 
capacity and roughly 70OMB of RAM. 


Ergonomics and Security of Graphical Email 
Clients 

e Erik Barwaldt 

We look at the ease of finding a way around 
Current graphical email clients by investigating 
ergonomics, security, and extensibility. 
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§ CronRAT Malware Targets Linux Servers 


Security researchers at Sansec (https://sansec.io/research/cronrat) have found a 
new stealth attack that targets Linux servers and uses a non-existent calendar 
day to stay off the radar. This remote access Trojan (RAT) masks the actions of 
the attack by using the date February 31 and targets Linux-based web stores to 
trigger online payment skimmer threats. 

The new CronRAT attack can execute fileless malware, launch malware in sep- 
arate subsystems, control servers disguised as Dropbear SSH services, hide 
payloads in legitimate cron tasks, and run anti-tampering commands. CronRAT 
bypasses browser-based security scans 
and has already been discovered tn live 
online stores. The threat was injected 
into servers via a Magecart (payment 
skimming) attack. 

This attack is made possible be- 
cause cron only checks for a date for- 
mat and not that the date of the task 
is legitimate. The crontab date specifi- 
cation for CronRAT is 52 23 31 2 3, 
which would generate a runtime error 
upon execution. However, that run- 
| time will never happen, because the 
Image © Ton Snoei, 123RF.com date doesn't exist. 

Once CronRAT Is executed, it contacts a Command and Control (C2) server at IP 
address 47.115.46.167:443 using a fake banner for the Dropbear SSH service. The 
payloads of the commands are obfuscated with multiple layers of compression and 
Base64 encoding. 

CronRAT is considered a serious threat to Linux e-commerce servers and has 
managed to bypass most detection algorithms. Sansec had to rewrite its algorithm 
to catch this dangerous threat. 





§ AlmaLinux OS 8.5 Now Available 


After CentOS dove into the “stream,” AlmaLinux (Attps:/almalinux.org/) has be- 
come one of the favorite replacements for the free take on Red Hat Enterprise Linux 
(RHEL). If users and businesses ever had any doubts about how well AlmaLinux 
would be able to keep up with RHEL, look no further than the release of 8.5 Stable 
as a bellwether on how well the fledgling platform will be able to keep up with the 
stalwart champ of enterprise Linux. 

Within 48 hours of the RHEL 8.5 release, AlmaLinux 8.5 Stable was made avail- 
able. This is the third stable release of the OS, which speaks to the commitment the 
AlmaLinux Foundation has made to deliver on its promises. 

The latest release of AlmaLinux includes improvements to container management 
tools, module streams, as well as enhancements and additions to System Roles. 
Other additions/improvements have been applied to the web-based GUI, Cockpit, 
OpenJDK (version 17 now available), per- 
sonal access tokens, Network Time Security © 
(NTS) for Network Time Protocol (NTP), 

SCAP Security Guide, Ruby (3.0), PHP 
(7.4.19), Node.js (16), NGINX (1.20), Squid 


(4.15), Mutt (2.0.7), GCC (11), LLVM 
(12.0.1), Rust (1.54.0), and new repositories 
for Resilient Storage and Plus. QO 
Find out more by reading the full release < 
notes (httops:/wiki.almalinux.org/release- | 





notes/8.5.htm/) and downloading AlmaLinux 
8.5 (https://repo.almalinux.org/almalinux/8/ 
ISOS/X86_64/). 
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NEWS 


SHE" Kernel News 


Zack’s Kernel News 





Chronicler Zack Brown reports 
on the latest news, views, 
dilemmas, and developments 
within the Linux kernel 
community. 


By Zack Brown 


Author 

The Linux kernel mailing list comprises 
the core of Linux development activities. 
Traffic volumes are immense, often 
reaching 10,000 messages in a week, and 
keeping up to date with the entire scope 
of development is a virtually impossible 
task for one person. One of the few brave 
souls to take on this task is Zack Brown. 
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Supporting New Hardware 
Features: UINTR 

Sohil Mehta from Intel brought up User 
Interrupts (UINTR), a hardware technol- 
ogy that allows delivering interrupts di- 
rectly to user space. Interrupts are a gen- 
eral-purpose idea in multitasking that 
have been around for decades. The idea 
is that if something on the system 
changes state, it might “interrupt” what- 
ever process is currently running, in order 
to handle this change. After that, process 
execution returns to normal. There are 
hardware interrupts, software interrupts, 
and a whole pile of tools and ideas re- 
lated to triggering and handling inter- 
rupts. Interrupts happen all the time. For 
example, when multiple programs are 
running at the same time, the system 
clock triggers an interrupt many times per 
second to switch between those processes 
and make it look like they are all running 
simultaneously. 

With UINTR, a particular interrupt tar- 
gets a particular process already running 
in user space. This way the interrupt can 
bypass time-consuming operations in the 
Kernel and go directly to where it is in- 
tended. Sohil said, “There is a ~ 9x or 
higher performance improvement using 
User IPI over other IPC mechanisms for 
event signaling.” And he went on to say 
that “Today, virtually all communication 
across privilege boundaries happens by 
going through the kernel. These include 
signals, pipes, remote procedure calls 
and hardware interrupt based notifica- 
tions. User interrupts provide the foun- 
dation for more efficient (low latency 
and low CPU utilization) versions of 
these common operations by avoiding 
transitions through the kernel.” 

Aside from all the normal sources of 
interrupts - the kernel or hardware de- 
vices — Sohil said that UINTR also intro- 
duces interrupts that originate from an- 
other user process called User IPIs. 

Because UINTR is a hardware feature, 
Sohil also added, “The first implementa- 
tion of User IPIs will be in the Intel proces- 
sor code-named Sapphire Rapids. Refer 
[to] Chapter 11 of the Intel Architecture 


instruction set extensions for details of 
the hardware architecture.” 

He also said, “I am also planning to 
talk about User Interrupts next week at 
the LPC Kernel summit,” and concluded, 
“We are hoping to get some feedback on 
the direction of overall software architec- 
ture - starting with User IPI, extending it 
for kernel-to-user interrupt notifications 
and external interrupts in the future.” 

Sohil also wanted feedback from the 
kernel developers about the future direc- 
tion Intel should take with this technol- 
ogy. For example, “Should Uintr inter- 
rupt all blocking system calls like 
sleep(), read(), poll(), etc?” And “Should 
the User Interrupt Target table (UITT) be 
shared between threads of a multi- 
threaded application or maybe even 
across processes?” 

An interesting aspect of UINTR, Sohil 
pointed out, is that it is not available to 
all user processes by default. As he put 
it, “User Interrupts (Uintr) is an opt-in 
feature (unlike signals). Applications 
wanting to use Uintr are expected to reg- 
ister themselves with the kernel using 
the Uintr related system calls.” 

There were also some security issues 
that Sohil wanted to point out. He said, 
“The current implementation expects 
only trusted and cooperating processes 
to communicate using user interrupts. 
[...] Currently, a sender can easily cause 
a denial of service for the receiver by 
generating a storm of user interrupts. A 
user interrupt handler is invoked with 
interrupts disabled, but upon execution 
of uiret, interrupts get enabled again by 
the hardware. This can lead to the han- 
dler being invoked again before normal 
execution can resume.” 

Sohil added, “To enable untrusted pro- 
cesses to communicate, we need to add 
a per-vector masking option through an- 
other syscall (or maybe IOCTL). How- 
ever, this can add some complexity to 
the kernel code. A vector can only be 
masked by modifying the UITT entries at 
the source. We need to be careful about 
races while removing and restoring the 
UPID from the UITT.” 


Dave Hansen replied to Sohil’s an- 
nouncement, suggesting some improved 
wording, and said, “Your problem in all 
of this is going to be convincing folks 
that this is a problem worth solving.” 

Dave also highlighted the 10x speedup 
as a real selling point for Sohil, and Sohil 
replied: 

“One thing to note, the 10x gain is only 
applicable for User IPIs. For other 
source[s] of User Interrupts (like kernel- 
to-user notifications and other external 
sources), we don’t have the data yet. 

“I realized the User IPI data in the 
cover also needs some clarification. The 
10x gain is only seen when the receiver 
is spinning in User space - waiting for 
interrupts. 

“If the receiver were to block (wait) in 
the kernel, the performance would drop 
as expected. However, User IPI (blocked) 
would still be 10% faster than Eventfd 
and 40% faster than signals.” 

Sohil posted a table of speed compari- 
sons between User IPI versus things 
such as pipes and signals, saying that 
the latency values were all “relative” to 
each other, rather than given in absolute 
quantities of time. 

To which Greg Kroah-Hartman replied: 

“Relative is just that, ‘relative’. If the 
real values are extremely tiny, then rela- 
tive is just ‘this goes a tiny tiny bit faster 
than what you have today in eventfd’, 
right? 

“So how about ‘absolute’? What are we 
talking here? 

“And this is really only for the ‘one us- 
erspace task waking up another user- 
space task’ policies. What real workload 
can actually use this?” 

Sohil responded with some absolute 
microsecond measurements - though 
he hedged a bit, saying, “The data here 
is more of an approximation with the 
final performance expected to trend in 
this direction. [...] The overall gain in 
a real workload would depend on how 
it uses IPC.” 

Sohil also responded to Greg’s question 
about what real-world workloads would 
use Intel’s feature. He replied, “User mode 
runtimes is one [of] the usages that we 
think would benefit from User IPIs. Also as 
Jens mentioned in another thread, this 
could help kernel to user notifications in 
io_uring (using User Interrupts instead of 
eventfd for signaling). Libevent is another 
abstraction that we are evaluating.” 


Earlier, Greg also pointed out that 
Intel’s hardware feature was limited to 
only a single CPU. And he asked, “Are 
syscalls allowed to be created that 
would only work on obscure cpus like 
this one?” 

To which Dave said, “Well, you have 
to start somewhere.” And he pointed to 
memory protection keys - another kernel 
feature that had a very narrow focus 
when it first went into the kernel. Dave 
said, regarding that feature, “At the point 
that I started posting these, you couldn’t 
even buy a system with this feature. For 
a while, there was only one Intel Xeon 
generation that had support. But, if you 
build it, they will come. Today, there is 
powerpc support and our friends at AMD 
added support to their processors. In ad- 
dition, protection keys are found across 
Intel’s entire CPU line.” 

And Dave concluded, “I encourage ev- 
eryone submitting new hardware fea- 
tures to include information about where 
their feature will show up to end users 
*and* to say how widely it will be avail- 
able. I’d actually prefer if maintainers re- 
jected patches that didn’t have this infor- 
mation.” 

To which Greg replied, “So, what are 
the answers to these questions for this 
new CPU feature?” But this question 
went unanswered. 

Meanwhile, Pavel Machek took note of 
the fact that this was an Intel CPU fea- 
ture and therefore was fundamentally 
implemented via CPU machine code 
(i.e., assembly language). Sohil’s origi- 
nal post listed about half a dozen assem- 
bly instructions on their CPUs that 
would be responsible for UINTR. And 
Pavel asked, “Are other CPU vendors al- 
lowed to implement compatible instruc- 
tions? If not, we should probably have 
VDSO entries so Kernel can abstract dif- 
ferences between CPUs.” 

This wasn’t answered, but several de- 
velopers had a technical discussion about 
potential use cases for UINTR. Ultimately 
the thread ended inconclusively - al- 
though clearly the Linux kernel develop- 
ers have a strong interest in supporting 
any CPU features that exist in the world 
and are not inherently insecure. 


Grooming Corporate 


Filesystem Maintainers 
The Paragon corporation has been the of- 
ficial maintainer of the NTFS3 filesystem 
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for several months. Considering this, 


there was an interesting conversation on 


the kernel mailing list recently. Konstan- 
tin Komarov from Paragon sent in a 
patch to update NTFS3, and Linus Tor- 
valds replied: 


“Well, I won’t pull until the next merge 


window opens anyway (about a month 


away). But it would be good to have your 


tree in linux-next for at least a couple of 
weeks before that happens. 


“Added Stephen to the participants list 


as a heads-up for him - letting him 


know where to fetch the git tree from will 
allow that to happen if you haven’t done 


so already. 

“The one other thing I do want when 
there’s big new pieces like this being 
added is to ask you to make sure that ev- 
erything is signed-off properly, and that 
there is no internal confusion about the 


GPLv2 inside Paragon, and that any legal 


people etc are all aware of this all and 
are on board. The last thing we want to 
see is some “oops, we didn’t mean to do 
this’ brouhaha six months later. 

“I doubt that’s an issue, considering 
how public this all has been, but I just 
wanted to mention it just to be very ob- 
vious about it.” 

The GPL issue is no joke and is one of 
the main reasons Linus adopted the 
whole “signed-off-by” procedure that ac- 
companies all patches these days. There 
were some accusations in the past about 


non-GPL code being included in the GPL. 


At the time, it was extremely difficult to 
trace the origins of each patch, so the 
dispute dragged on for quite awhile. 
Finally, Linus decided that each patch 
needed to be signed off by developers 


and reviewers, in part for that reason - to 
make sure that the person submitting the 


patch affirmed that it was being submit- 
ted under the terms of the GPLv2. Then, 
any future issues could be resolved by 
simply querying the patch history. 

On another level, when Linus made 
this open reminder to the Paragon peo- 
ple, he might have been trying to pre- 
empt any future problems where Para- 
gon might claim they had not been in- 
formed of the rules. 
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But on a less legal level, Linus seems 
to simply be training a new contributor 
in the ways of developing the kernel and 
submitting patches - making sure every- 
thing is GPLed, timing their submissions 
with the merge windows, and so on. 

Konstantin thanked Linus for the 
info and sent the patch along to Ste- 
phen Rothwell to include in the linux- 
next tree. 

And Konstantin confirmed, “Indeed, 
there is no internal confusion about the 
GPLv2 and we mean to make this con- 
tribution.” 

Stephen replied to Konstantin, saying: 
“Thanks for adding your subsystem 
tree as a participant of linux-next. As you 

may know, this is not a judgement of 
your code. The purpose of linux-next is 
for integration testing and to lower the 
impact of conflicts between subsystems 
in the next merge window. 

“You will need to ensure that the 
patches/commits in your tree/series have 
been: 

e submitted under GPL v2 (or later) and 
include the Contributor’s Signed-off-by, 

e posted to the relevant mailing list, 

e reviewed by you (or another main- 
tainer of your subsystem tree), 

e successfully unit tested, and 

e destined for the current or next Linux 
merge window. 

“Basically, this should be just what you 
would send to Linus (or ask him to 
fetch). It is allowed to be rebased if you 
deem it necessary.” 

So Stephen too, in the same spirit, 
seemed to be training the Paragon 
folks on proper GPL procedures and 
other code submission practices, as 
well as managing their expectations 
about what they’d be likely to see with 
their code in the linux-next tree in the 
near future. 

Kari Argillander also did some of the 
same, reminding the Paragon folks to 
“add reviewed-by tag and signed-off-by 
tag” and to post the code as a plain-text 
patch to the mailing list for review by 
other developers. 

Linus posted again at a certain point, 
with further gentle instruction on how to 


get code into the kernel after marinating 
first in linux-next: 

“Ok, so ’'ve merged the biggest pieces 
of this merge window, and I haven’t actu- 
ally seen a NTFSv3 pull request yet. 

“IT wonder if you expected that being in 
linux-next just ‘automatically’ causes the 
pull to happen, because that’s not the 
case. We often have things ‘brewing’ in 
linux-next for a while, and it’s there for 
testing but not necessarily ready for 
prime time. 

“So linux-next is a preparatory thing, 
not a ‘this will get merged’ 

“So to actually merge things, I will ex- 
pect to get an explicit pull request with 
the usual diffstat and shortlog, to show 
that yes, you really think it’s all good, 
and it’s ready to merge. 

“Don’t worry about - and don’t try to 
fix — merge conflicts with possible other 
work that has been going on. Stephen 
fixes it for linux-next and makes people 
aware of it, and I want to _know_ about 
them, but I will then handle and re-do 
the merge conflicts myself based on what 
I have actually merged up to that point. 

“And of course, the other side of that is 
that if linux-next uncovered other issues, 
or if there are things holding things up, 
please _don’t_ feel obligated to send me 
a pull request. There is always the next 
merge window.” 

And that was the end of the discus- 
sion. It may seem like a surprisingly soft 
kid-glove approach, but in fact, for all 
the corporations that contribute code to 
the Linux kernel, it can be surprising 
how frequently a company doesn’t seem 
to know its ass from its elbow in terms 
of working with kernel developers. 

Apparently instead of allowing each 
case of “first contact” to crash and burn 
for awhile, while their engineers and 
marketers and lawyers bark conflicting 
internal orders and end up looking ever- 
more absurd in public, Linus and others 
are taking the approach of trying to 
make the initiation as smooth and un- 
surprising as possible. 

It seems like a much better approach 
than some of the freaky scenarios that 
have come before. BEE 
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The Buzz on Fuzz 


Fuzzing is an important method for finding bugs and security vulnerabilities in software. Read on to 
find out what fuzzing is and which methods are commonly used today. 


By Andreas Zeller 


t was a dark and stormy night. Bart Miller - you’ll find an 

interview with him at the end of our feature - was work- 

ing at home, connected by 1200-baud modem to the Uni- 

versity of Wisconsin’s mainframe computer. But every 
thunderclap meant that something went wrong: the lightning 
strikes disrupted data transmission over the phone line and 
garbled individual characters, forcing Miller to start over time 
and time again. 

Each time he restarted, he noticed how many programs 
couldn’t cope with disrupted data - they crashed, hung up, or 
otherwise stopped working in some uncontrollable way. 
Shouldn’t programs do much better with invalid or glitched 
input? Miller decided to have his students systematically inves- 
tigate this problem and gave them a programming assignment. 

That night in the fall of 1988 is considered the birth of fuzz 
testing, by far the most important method today for testing pro- 
grams for robustness and checking for security vulnerabilities. 
Professional programmers routinely use fuzzing to check for 
problems that could occur in the wild and might not be easy to 
anticipate. However, fuzzing is still a mystery to many part- 
time programmers and advanced users who program infor- 
mally (including many in the Linux community). This month 
we take a close look at fuzzing and why it is so important. 


What Is Fuzzing? 

The nature of fuzzing is revealed directly from Bart Miller’s 
programming assignment: “The goal of this project is to evalu- 
ate the robustness of various Unix utility programs given an 
unpredictable input stream. This project 

has two parts. First, you will build a fuzz 
generator. This is a program that will out- 
put a random character stream. Second, you 
will take the fuzz generator and use it to at- 
tack as many Unix utilities as possible, with 
the goal of trying to break them.” 

This programming task summarizes the 
basic idea of fuzzing: You automatically gen- 
erate random input, check to see if the pro- 
grams fed with it then do unpredictable 
things, and repeat these two steps very often 
and very quickly. 


out = ene 


return out 
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import random 


"""Generate a 
in the range 


string length 


In the process, fuzzers use various techniques to find errors. 
Purely random input is easy to generate; it finds errors in input 
processing, such as buffer overflows. Model-based fuzzers use 
grammars and other language models to generate valid and tar- 
geted input. Evolution-based fuzzers mutate test input to find 
variants that cover as much code as possible. Constraint-based 
fuzzers can solve complex constraints in program code, but 
they take a long time to do so. 

Fuzzing with purely random input is very simple to do: A 
few lines of program code are all that is needed to generate 
the necessary input. For example, the fuzzer() function from 
Listing 1 generates strings of random characters that look 
something like this: 


1 7#%"KHO=)$;%6*; >638:*>80"=</>(/* 
:-(2<4 !:5*6856&2""11<74%<%7, 4.84 


Bart Miller referred to this kind of input as fuzz, meaning 
unstructured, random data. What can fuzz do? Let’s imagine 
that you feed this string to a program that expects a five- 
digit postal code. If it has a buffer of five characters for the 
input, but the actual input exceeds its capacity (the fuzz 
above is more than 60 characters), a buffer overflow can 
occur. Memory areas beyond the five-character buffer are 
overwritten by the input in a more-or-less random manner, 
which, in turn, causes the program to behave more-or-less 
randomly. It could crash or enter an infinite loop when try- 
ing to read the five digits. 


Listing 1: Simple Fuzzer in Python 


def fuzzer(max_length=100, char_start=32, char_range=32): 


string of up to ‘max_length’ characters 
[ charrotart = caarestart =.) charmerance: ads) s 2 


= random.randrange(0, max_length + 1) 


for i in range(0O, string length): 


out += chr(random.randrange(char_start, char_start + char_range) ) 





If the program receives its input via a web page, attackers 
could, for example, enter a string like the one above into a 
form and thus attempt to disrupt the program or render it un- 
usable. Because you can generate millions of fuzz input items 
like this every minute, the attackers would also have many at- 
tempts to inject fuzz. All they would have to do is sit back and 
let the fuzz generator do its work; after hours or days, a crash 
might occur. 

It doesn’t have to stop at the crash, however. Since buffer 
overflows can also overwrite critical data such as passwords or 
even program code, it may be possible to design the input in 
such a way that the attacker gains control of the program or 
even the computer. This part of the work is not yet as auto- 
mated as fuzz testing; but, if successful, huge rewards beckon 
for the discovered vulnerability. 

Today’s programs are protected against such attacks. As a 
general rule, you should not trust any data that is under the 
control of a third party (i.e., that comes from users, other com- 
puters, or other programs). A web application that expects a 
five-digit postal code should therefore make sure that the input 
actually consists of five digits by already checking the input 
form to see whether the input is correct. The server needs to 
check the transmitted data for correctness, and all programs 
that process it downstream should do the same. Last but not 
least, programs on the network should only allow a limited 
number of failed attempts before blocking access. 

In 1988, such mechanisms were uncommon, and what Mill- 
er’s students found was alarming: They could crash more than 
a third of all Unix utilities within seconds by hitting them with 
random input. Imagine what would happen today if a third of 
all Web applications were vulnerable in such a trivial way - the 
Internet as we know it would be taken over in seconds. 

In 1988, however, the Internet was still in its infancy, and 
every administrator knew the users on their machines per- 
sonally. In fact, Miller initially had trouble getting his find- 
ings published. The typical response was, “So what? Why 
should I care what happens to invalid data? My users 
send me valid data!” The open source developers of the 
time saw it differently and quickly adapted their programs 
to identify invalid input and reject it in a controlled manner. 
In this way, most GNU programs and the Linux kernel 


quickly developed resistance to fuzz input, and the standard 
thus slowly established itself in the rest of the programming 
world. 


Automatic Testing 

Once you have hardened a program against random Miller- 
style strings, it is very hard for a fuzz generator to find errors. 
This is because most randomly generated input is invalid. For 
example, suppose our program processing a postal code 
shows an error for the postal code 00000, perhaps because 
that number stands for a particular address. What is the 
chance of getting this input by chance? If the (simplified) fuzz 
generator generates input between one and 100 characters, 
then the chance that it will be five characters is 1:100. Let’s 
also assume that the generator generates 10 different (print- 
able) characters, including 10 digits. Then the chance that 
five digits will come out is 1:10° (that is one in 100,000). The 
chance for five zeros is even only 1:1005 (that is one in 10 bil- 
lion). Multiplied by the chance that we get any input of 
length five at all, that’s one in one trillion. Modern web ap- 
plications work fast, but even if we assume one millisecond 
per interaction, in the worst case we would need 31 years of 
continuous computing to discover the error. There’s more to 
be gained from mining bitcoins. 

However, if you know how the input is constructed, you can 
drastically increase the chance by making that knowledge 
available to the fuzz generator. This is where a more powerful 
class of generators comes into play, whose operating principle 
dates back to 1972. Model-based fuzzers use a specification of 
the input format to generate valid inputs a priori, bypassing the 
numerous failed attempts with purely random strings. 

One well-known and well-understood method for specify- 
ing input formats is grammars that define the structure of 
the input. For example, the grammar from the box “Struc- 
ture of a Postal Code” describes the structure of a postal 
code. Such a grammar consists of rules that specify the 


Cu P c Pp if Tl 
Cte eri iran AT Aorks fC nrnnna 
Structure OT a FOSTal LOdGe 


<pOSsuccde> 5= <disit> diet <dieit><digit><dasit> 


<digit> ==0 |i )2|)3s ea )s |}el)7)2e\9 
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Fuzz Testing 


structure of a single input element: A <postcode> consists of 
five directly consecutive digits; a digit is one of ten alterna- 
tives separated by |. 

A model-based fuzzer now uses this grammar to generate 
input. To do this, it starts with an input element on the left 
(such as <postcode>) and replaces it with the text on the right 
(<digit><digit><digit><digit>). It repeats the whole thing until 
all symbolic input elements are replaced. If there are several al- 
ternatives to choose from on the right-hand side, it picks one of 
them at random. Thus, the fuzzer replaces each <digit> input 
element with one of the digits @ to 9, so that it generates, say, 
the random sequence 43672. It could also be 34829 or 12456 - 
the important thing is that it is five digits. 

Using this grammar, the chance of finding the problematic zip 
code 00000 can be reduced to 1:100,000. This still sounds like a 
lot of testing, but even model-based fuzzers generate millions of 
inputs within minutes. And unlike purely random input, their 
input is always valid, so they can go deeper into the program 
and thus find logical errors beyond input processing. 





Constraint-Based Fuzzer 


Evolution-Based Fuzzer 





Model-Based Fuzzer 





Random Entries 


Input Processing 














Figure 1: Different fuzzer types are able to dig down 
to different depths in the software. 


Grammar for Addresses 


<address> := <first_name> <name>, <street> <house_number>, 


<first_name> := Anton | Berta | .. | <random_name> 
<name> := Mueller | Schmidt | .. | <random_name> 


<city> := Berlin | Munich | <random_name> 


<street> := Main Street | Putzbrunner Street | <random_name> Street 


<house_number> := <digit> | <house_number><digit> 


<random_name> := <letter> | <random_name><letter> 


Cletien> = = 7 eB pec i). | oe 


SQL Injection Via a Grammar 


<city> := Berlin | Munich | <random_name> | <SQL_injection> 


<SQL_injection> := ‘); DROP TABLE addresses; 
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<postcode> <city> 


Where Fuzzers Find Errors 

Although input processing generally rejects purely random 
input, model- and evolution-based fuzzers can more easily dig 
down into the program logic. Constraint-based fuzzers take this 
up a notch (Figure 1). In fact, model-based fuzzers easily can 
be used to generate even complex input. The grammar from 
the box “Grammar for Addresses,” for example, generates ad- 
dress data. 

A record like Berta Mueller, Main Street 10, 43679 Munich is 
generated from the <address>. The rules for <house number> and 
<random_name> generate a sequence of digits and letters, respec- 
tively, resulting in more unusual records like Anton GJK, EI- 
UYLHK Street @@, 23165 Berlin. 

Since the user can create and extend grammars, they allow 
the fuzzer to be more targeted. For example, a security tester 
could add SQL injection to the preceding grammar, that is, an 
attack that injects database commands into an input (see box 
entitled “SQL Injection Via a Grammar”). 

This injection adds SQL commands to the addresses, which can 
lead to the deletion of data if the inputs are not checked properly. 
At this point, at the least, it becomes clear that fuzzing experi- 
ments should only be carried out in a controlled environment on 
your own computer. In some jurisdictions, if you try to fuzz a 
third party, you risk being prosecuted for data manipulation. 

Grammar-based testing is widespread in practice, but it is 
primarily used to test your own programs - not least because 
the grammar can always be adapted to concrete test goals. 
Mozilla and Google, for example, have found and fixed thou- 
sands of problems in JavaScript interpreters using the gram- 
mar-based Langfuzz [1], which generates JavaScript pro- 
grams. Csmith [2], a program for generating valid C pro- 
grams, is used extensively in compiler tests. Model-based For- 
matFuzzer [3] specializes in binary input such as image files 
or archives. 

If you switch to a browser after reading this, you can rely on 
its security - thanks in no small part to model-based fuzzing. 


Successful Thanks to Mutation 
For all their power, model-based fuzzers have one decisive dis- 
advantage: You first have to write a grammar for them. For rare 
or application-specific input formats, this requires considerable 
overhead, especially if you first need to understand the format. 
However, if you have sample input data, you 
might wish to try the youngest class of test 
generators: evolution-based fuzz generators. 
Evolution-based fuzz generators also aim 
to produce the most valid input possible in 
order to get deep into the program. However, 
they achieve this goal not by using an input 
description such as a grammar, but by sys- 
tematically changing (mutating) existing 
input. For instance, consider a German ad- 
dress: Putzbrunner StraBe 71, 81739 Munich. 
A mutation might consist of swapping two 
characters: Puztbrunner StraBe 71, 81739 Mu- 
nich. Alternatively, you can replace one 
character with another (Putzbrunner Straxe 
71, 81739 Munich) or delete a random charac- 
ter (Putzbrunner StraBe 71, 8173 Munich). 


Each mutation thus delivers new input to test. The range of 
variation is not as high as with purely random input or with 
model-based testing, but the probability of valid input is still 
high. In the preceding examples, for instance, deleting just 
one digit from the postal code leads to an invalid address. 

But the real trick with evolution-based fuzzers is that they 
use mutations to systematically evolve a set of successful in- 
puts. The goal is to cover as much program behavior as possi- 
ble. For this purpose, fuzzers maintain a set (known as a pop- 
ulation) of particularly promising input. They start with a set 
of known and valid input, which they mutate, increasing the 
population. 

The fuzzer now measures which places in the program the 
input reaches (coverage). Input that reaches previously uncov- 
ered locations is considered particularly promising, is retained, 
and serves as the basis for further mutations. In contrast, input 
that does not reach new locations is dropped from the popula- 
tion (selection). In this way, the population continues to evolve 
with respect to the goal of reaching as many program locations 
as possible and thus covering as many program behaviors as 
possible. 

In the previous sample mutations, for example, the last 
mutation (8173) would be classified as promising because it 
is the first to reach the error-handling code with the four- 
digit postal code and thus would expose new program behav- 
ior. The mutation from Putzbrunner to Puztbrunner, on the other 
hand, is unlikely to cover new code; it would be removed from 
the population in the long run. 
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What about the chance of generating the special postal code 
000002 If the code is structured to query the postal code bit by 
bit like in Listing 2, as each partial condition is met, another 
piece of code is executed that checks the next digit. After nu- 
merous runs, 81739 mutates first to 01739, then to 00739, then 
to 00039, and finally to 00000, so that again more code is dis- 
covered. In fact, evolution-based fuzzers are also able to detect 
partial success when comparing strings and numbers, so that 
they slowly work their way towards a goal through more and 
more mutations. 

Evolution-based fuzzers resemble natural evolution - many 
failed tests, but effective in the long run. Most importantly, 
they are frugal: Test input is often available or can be gleaned 
from concrete sequences; the coverage of a test is easy to mea- 
sure. Moreover, evolution-based fuzzers do not require further 
knowledge about input formats. Their major drawback is their 
dependence on good starting input: If the initial input does not 
contain a certain feature, it is unlikely that the feature will ever 
be generated. 

The classic evolution-based fuzzer is American Fuzzy Lop 
(AFL [4]), a highly optimized fuzzer that has been able to 
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Fuzz Testing 


detect numerous bugs in existing programs. The majority of 
security-critical Linux programs are now tested around the 
clock by AFL and its offshoots, such as in Google’s OSS Fuzz 
project. 

Although AFL and its offshoots are primarily used in “hos- 
tile” scenarios where the program authors are not expected to 
cooperate, Sapienz [5] is an evolution-based fuzzer that Face- 
book uses to test its own programs and applications. Anyone 
who uses the command line on a Linux system or opens the 
Facebook app on their cell phone will benefit from the work of 
evolution-based fuzzers. 


Detecting Conditions 

The fourth class of fuzz generators also aims to reach as 
many places in the program code as possible. Instead of the 
trial-and-error principle that evolution-based fuzzers use, it 
relies on systematic recognition and resolution of path condi- 
tions. This refers to conditions under which execution takes a 
particular path in the program and thus reaches a particular 
code location. 

In these constraint-based fuzzers, specialized constraint solv- 
ers are used — programs that automatically seek a solution for a 
given set of conditions. For a zip code constraint like the one 
described previously, a constraint solver produces the solution 
00000 within hundredths of a second. 

However, constraint solvers show their true strength in com- 
plex arithmetic calculations. If the condition is something like 
the one in Listing 3, a constraint solver will also determine 
suitable values for x and y in a very short time. Grammar-based 
fuzzers, on the other hand, still rely on chance, and evolution- 
based fuzzers need to hope that their starting population con- 
tains suitable values for x and y. Such strengths are of particu- 
lar interest when deep conditions of program logic must be sat- 
isfied. But constraint solvers can also quickly solve non-trivial 
properties of input files such as checksums. 

Constraint solvers are powerful, but not omnipotent. If it is 
necessary to recover the original from an encrypted text, even a 
constraint solver very, very often cannot do more than guess 
the key. Moreover, constraint solvers work slowly. Grammar- or 
evolution-based fuzzers can sometimes test thousands of input 
items and - depending on the complexity of the constraints - 
end up getting to home base faster than a constraint solver by 
making clever guesses. 

KLEE [6], a very popular constraint-based fuzzer, has found 
hundreds of program errors in C programs. Microsoft uses its 
own constraint-based fuzzer named SAGE [7], which uses 
grammars for Office documents to systematically search for 
bugs in office software. It has saved Microsoft and its users 
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hundreds of millions of dollars of damage. NASA also relies on 
constraint-based fuzzers to fully test any situation a spacecraft 
might encounter [8]. 


Outlook 


Modern fuzzers have long since stopped relying on just one 
technique and have resorted to pragmatic combinations. In 
the end, the fuzzer that finds the worst errors the fastest is 
the right choice. For example, it makes sense to use simple 
random strings to look for errors in input processing first, 
before a grammar or mutated test inputs target errors at 
deeper levels. 

Grammars, evolution, and constraints can be combined in a 
variety of ways. Which combination is most successful under 
which conditions is the subject of heated debate among fuzz- 
ing researchers and developers. The user is less interested in 
which variant of which tool is used - the most important thing 
is to get started with fuzzing in the first place. Software that 
has never been exposed to a fuzzer is very likely to have errors 
that can be triggered by random input. 

For C programs, fuzzers such as AFL or KLEE promise quick 
results; a grammar for simple input is created in next to no time. 
But for developers and attackers alike, if you don’t use fuzzing 
yourself, you risk others doing so and quickly exploiting the 
bugs they find. However, once you have set up a fuzzer, you can 
sit back and relax: From here on, the fuzzer takes over, tirelessly 
testing with new input until the red light comes on, indicating 
that a bug has been found. Then the troubleshooting and debug- 
ging begins — how to automate that is another story. Hmm 


Info 

[1] Langfuzz: https:/github.com/mozilla/JSBugMon/blob/master/ 
langfuzz.py 

[2] Csmith: https:/embed.cs.utah.edu/csmith/ 

[3] FormatFuzzer: https:/github.com/uds-se/FormatFuzzer 

[4] AFL: https:/github.com/google/AFL 


[5] Sapienz: 
https://engineering.fb.com/20 18/05/02/developer-tools/ 
sapienz-intelligent-automated-software-testing-at-scale/ 
[6] KLEE: https:/klee.github.io 


[7] SAGE: 
https://patricegodefroid.github.io/oublic_psfiles/ndss2008. pdf 


[8] “Fuzzing NASA Core Flight System Software”: 
https:/www.youtube.com/watch ?v=D5yillMy2Lg 
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Talking Fuzz Testing with Bart Miller 
By Jens-Christoph Brendel 


We asked fuzzing pioneer Bart Miller for his thoughts 
on fuzz testing and the role it plays in software 
development. 


Linux Magazine: Are some applications more suitable 
than others for fuzz testing? 


It’s not a question of suitability. It’s a question of iden- 
tifying the interface to the application and figuring out 
how best to test against that interface. For example, 
some programs are very simple and take their input 
from a file or keyboard (“standard input”). Generating 
input for such a program is pretty simple. Other pro- 
grams might use the mouse and keyboard, which 
means their input comes from the window system 
(usually X Window in Unix/Linux, Win32/Win64 in Mi- 
crosoft Windows, or Aqua in macO$). This type of 
input requires the test program to identify the win- 
dows and their sizes and then generate proper win- 
dow events (mouse movements, key up, and key 
down events). A phone app has input based on 
touches and gestures, so you have to be able to ran- 
domly generate these events. 


LM: What if one uses fuzz tests for security purposes? 
An attacker uses sophisticated techniques to overcome 
a victim’s security measures. How likely is it that a ran- 
dom input string will match such a precisely con- 
structed attack? 


It's the other way around. An attacker often uses fuzz 
testing to see if they can crash a program. This means 
that they can cause the program to enter a state that the 
programmer didn't intend. In the intelligence commu- 
nity, they call this “owning the bits” ... 1.e., owning bits 
in the program state that you should not be able to own 
(for example, overflowing a buffer or causing a number 
to be negative when it should always be non-negative, 
such as an array subscript). 


Once an attack (or analyst) causes a crash, they know 
that the program is not checking everything that it 
should check. So, now the task is to craft your input in 
such a way so as to exploit this reduced checking. 


LM: Does a fuzz test possibly reveal rather simple errors? 


If the application crashes due to a gross violation of its 
expectations, might errors remain undetected that only 
occur at the end of a chain? 


The random nature of fuzz testing means that you are 
randomly walking through the state space of the pro- 
gram. Pure random input (“unstructured” fuzz testing) 
is likely to trigger errors earlier in processing and not 
cause execution so deep into the program’s logic. 
However, it might and sometimes does find more 
complex logic errors. 
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You can add structure to your input: for example, prop- 
erly structured window events or properly structured 
network packets. These will have random contents but 
valid structure so will allow testing of deeper structures 
in the code. 


The key to any test regime is “coverage” — in other 
words, how much of the program’s code have you actu- 
ally tested? Many modern fuzz testers, like AFL, libfuzz, 
or HongFuzz are “coverage guided” fuzz testers. They 
track which blocks of code were executed for each input 
and then try to generate new inputs that will test other 
parts of the program’s code. 


LM: How confidently can one infer from a discovered 
bug that a class of bugs might be hiding behind it? 


Certainly a human analyst, when they find a specific type 
of bug, should look elsewhere in the code for the same 
class of bug. | haven't seen any good way to automate 
this idea. 


LM: Should fuzz testing always go hand in hand with de- 
bugging of detected errors to uncover the exact mecha- 
nism leading to the error? 


This is a good question and not just for fuzz testing. 
When you find erroneous behavior in your program, 
you have not found the bug until you can identify the 
specific line(s) of code that caused the behavior and 
explain why the behavior happened. If you cannot do 
this, then you are just guessing about the fix. It might 
appear that the bug went away, but you have likely 
just disguised it or moved it. No good programmer 
will be happy until they have found the cause. 
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Parrot OS offers a more secure desktop with practical tools for both newbies and veteran users that 
encourage better security habits. By Bruce Byfield 


he Parrot OS home page [1] lists 
four major concerns: security, 
software freedom, a lightweight 
system, and cross-platform por- 
tability. To these concerns, it also adds 
a thorough development stack and the 
goal “to push newbies into good hab- 
its.” Of all these concerns, Parrot is 
best known for being security con- 
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scious. However, it succeeds in all of 
these concerns to a degree. In particu- 
lar, by making privacy and security 
tools part of the standard install, Parrot 
is probably most successful in pushing 
all users - not just newbies - into bet- 
ter security habits. 

Based on Debian Testing, Parrot OS is 
a rolling release, with packages updated 





as soon as they have been tested, al- 
though official releases are periodically 
released as well. It also offers a choice 
of MATE or KDE Plasma as a desktop 
environment. Parrot requires 256MB of 
RAM, making it lightweight by modern 
standards and suitable for older systems 
as well as new ones. Its goal of cross- 
platform portability is seen in the num- 
ber of downloads available. The Home 
Edition is for general users (Figure 1), 
and the Security Edition is for penetra- 
tion testing and other security work and 
programming (Figure 2). Both the 
Home and Security Editions run virtu- 
ally in Boxes and include images for 
VMware and VirtualBox. The Security 
Edition includes variants for IoT and 
cloud appliances. Parrot OS also offers a 
Pwnbox version that runs in a web 
browser and an ARMv7 architecture 
version. In addition, the installation im- 





ages for the Home and Security Editions < 


include a Live version. Installation is 
via the Calamares installer (Figure 3), 


one of the simplest installation methods =< 


available for distributions. 


Available Applications 
Parrot OS’s most notable feature is its 
software selection. Only a handful of 


productivity applications are available, E 


Distro Walk — Parrot OS Ez 


such as LibreOffice and Gimp, in keep- Applications 
ing with the basic security principle of 
installing the minimum software re- 

quired in order to reduce the opportu- 
nities for possible attacks. Beyond = 


Privacy 


Keita: 





these basics, users should install only ae igen | ¥ asc2log 
the software they require. Even the @ sound & video Y canbusload 
game menu is limited to a 2D chess @ cames d ‘¥ can-calc-bit-timing 
game - a hint, perhaps, at the intellec- Pentesting Most Used Tools ¥ candump 
tual users that Parrot expects to attract. @ Programming Information Gathering Y canfdtest 
Both editions have the usual MATE ® system Tools Vulnerability Analysis ¥ cangen 
system tools, such as BleachBit for System Services Web Application Analysis Y cangw 
cleaning up installed software and he Accessories Exploitation Tools ¥ canlogserver 
GParted for disk partitioning, but only Universal Access Maintaining Access h cto ba 
the Security Edition includes a menu “ Post Exploitation ae 
; : : 7 ‘ cansniffer 
for starting and stopping services. Sim- Password Attacks . 
ilarly, both editions include Vim and Wireless Testing ~ isotpdump 
j 14 7 isotpperf 
Neovim. However, the Home Edition Sniffing & Spoofing 7. 
includes only three dedicated program- | | pies 
' , Digital Forensics 4 isotpsend 
ming tools: Geany, the programming 
. ' : Automotive Can Utils > isotpserver 
text editor; links to VSCodium, a col- ) 
: : : i Reverse Engineering Y isotpsniffer 
lection of open source binaries for Mi- ~ 
crosoft’s Visual Studio Code; and Zeal, Reporting Tools , isotptun 
a help site for developers. By contrast, : ao 
the Security Edition includes an addi- < vicina 
tional five applications and links, ay 
among them Git Cola, a GUI for Git; Figure 2: The Security Edition contains dozens of scripts and apps, as 
Meld, a diff and merge tool; and shown by the extensive menu for penetration testing. 
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Figure 3: Parrot OS uses the Calamares installer. 


git-dag, a graphical depiction of Git 
history. From the selection of tools 
available in the two editions, the as- 
sumption seems to be that hobbyist 
programmers will use the Home Edi- 
tion, and administrators and more ad- 
vanced programmers working on large 
projects, often with remote versioning 
repositories, will install the Security 
Edition. 

However, it is in privacy and security 
tools that Parrot OS really stands out. 
Some of these tools, such as the Elec- 
trum Bitcoin Wallet, the Ricochet chat 
app, or the GNU Privacy Assistant, are 
developed by other projects; others, such 
as OnionShare or AnonSurf, are devel- 
oped by the Parrot Project itself. Occa- 
sionally, the available tools overlap. For 
example, Parrot OS offers both Tor and 
J2P for anonymous web browsing, leav- 
ing users to decide which one to use. 





AnonSurf is running 


Restart Tor Stats 


Change 
Identity 


Stop My IP 


Details ay) 











Figure 4: AnonSurf is an example 
of the simple efficiency of the apps 
designed by the Parrot Project. 
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The Home Edition focuses on pri- 
vacy. AnonSurf takes Tor one step fur- 
ther by anonymizing all system com- 
munication, beginning by shutting 
down potentially vulnerable apps and 
changing a user’s IP address with a 
single click of a button (Figure 4). Also 
included are five different encryption 
tools as well as a metadata cleaner and 
a secure file deleter. Unlike many pri- 
vacy tools, most of the tools in the 
Home Edition have easy-to-use graphic 
interfaces and embedded help that as- 
sume minimal prior knowledge. When 
a tool lacks a GUI, Parrot OS’s menu 
opens a terminal at the appropriate 
man page. As a result, more users 
should be encouraged to use these 
tools, and, perhaps, feel confident 
enough to learn more about the princi- 
ples behind them. 

The Security Edition’s menu is domi- 
nated by the Pentesting (penetration test- 
ing) top-level menu for forensic investi- 
gations. The menu is so crammed with 
scripts and apps that the second-level 
menu contains 14 items, including Most 
Used Items, Information Gathering, Vul- 
nerability Analysis, Exploitation Tools, 
Password Attacks, Sniffing & Spoofing, 
and Reverse Engineering, to say nothing 
of third- and fourth-level menus, some of 
which are even longer. The entries alone 
are the start of an education - who 
knew, for example, that Linux was im- 
portant enough in the car industry that 
there would be an Automotive option? 

All this is just an overview. Just listing 
the scripts and apps included in Parrot 


OS would take at least 5,000 words and 
fully documenting everything would 
take a book. Parrot OS essentially pro- 
vides a curated range of Linux privacy 
and security tools. Just having all these 
resources on one menu makes learning 
about security easier. 


Security Accessibility 
Security often means a trade-off with 
convenience, and Parrot OS is no ex- 
ception. To start with, KDE’s Plasma is 
hardly in keeping with the goal of 
being lightweight. For that matter, any 
desktop environment is just one more 
place where things can go wrong. For 
that reason, many security experts pre- 
fer to work exclusively from the com- 
mand line. In addition, while Debian 
Testing is often said to be more reliable 
than most distributions’ general re- 
leases, Debian Stable would be a more 
secure choice for building a distribu- 
tion. These choices seem to be made 
for the simple reason that many users 
prefer a desktop environment and to 
offer users apps closer to the latest re- 
leases (Debian Stable is sometimes 
several releases behind). 

If Parrot OS does not offer the most se- 
cure distribution possible, it does offer 
more security than most distributions. 
Parrot OS installs with a wide range of 
privacy and security tools, many of 
which are easily configured or installed 
ready for use. More than anything, Par- 
rot OS offers easy access to privacy and 
security. Although Parrot is potentially 
more secure than most distributions, its 
greatest accomplishment is security edu- 
cation, something that is sadly lacking 
elsewhere despite years of concern and 
growing necessity. Parrot OS provides 
both a basic curriculum and a practical 
set of tools. HEE 


Info 
[1] Parrot OS: https:/parrotsec.org/ 
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PDFs, the preferred format for file sharing, only offer primitive privacy and security measures. With 
these command-line tools, you can help your PDFs meet modern security requirements. By Bruce Byfield 


irst released in 1992, the Portable 
Document Format (PDF) shows 
no signs of disappearing. The for- 
mat has become a business stan- 
dard and the preferred format for send- 
ing files to a print shop. Closely related 
to PostScript (PS), PDF has the advan- 
tages of ensuring that files are seen ex- 
actly as the user intended and of elimi- 
nating concerns about the supported for- 
mats or available fonts on the recipient’s 
word processor. Yet for all these advan- 
tages, the PDF format comes from an era 
less concerned with privacy and secu- 
rity. It does include privacy and security 
permissions, but these are primitive by 
modern standards. In fact, these tools 
are literally a joke. As I have heard sev- 
eral times, PDFs exist in one of two 
states: compromised and about to be 
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compromised. Fortunately, the means 
exist to address issues that the PDF 
format itself does not. In addition to the 
countless scripts available for editing 
PDF content and structure, there are also 
a growing number that enhance privacy 
and security. 

PDF password protection continues 
to be used for the simple reason that it 
is widely available. LibreOffice users, 
for example, can set it by selecting File 
| Export As | Export As PDF... | Secu- 
rity. From this path, a password can be 
set to open a PDF file, as well as grant 
permission for if and how the file can 
be printed, edited, or copied. These 
settings will control a PDF file’s use by 
unsophisticated users, but they are no 
match for modern cracking tools. Aside 
from the laxness with which many 
users handle security, if the file is read 
in an environment in which the reader 
can control permissions (i.e., in most 
modern operating systems), the pass- 
word protection is easily and quickly 
bypassed [1]. LibreOffice itself pro- 
vides stronger protection with the op- 
tion of using a digital signature or a 
personal GPG key by selecting File | 
Export As | Export As PDF... | PDF Op- 
tions | Digital Signatures (Figure 1). 


—— 
+ S| 
‘_aceoenmasidaei! 


While this option guarantees the send- 
er’s identity, it does not cover every 
circumstance. 

Depending on your purpose, you may 
want to use one of the scripts found in 
the repositories of Debian and other 
major distributions. Some of these 
scripts can carry out numerous 
functions, but here I will only detail their 
privacy and security functions. Note that 
several have no man page, instead 
offering only a brief help option, which 
should be enough to figure out their use. 


pdfcrack 

Whenever passwords are used, some 
users are bound to forget them. These 
users provide a legitimate reason for an 
administrative tool such as pdfcrack [2], 
although these tools are, of course, open 
to abuse. You can use pdfcrack in two 
ways: with a search string using the 
option --charset STRING (-c STRING) 
(Figure 2) or with --word]ist=FILE 
(-w=FILE). Using a wordlist - any 
number of which are available online - 
is generally the most efficient, unless 
you have some idea of what the 
password might be, and you can use 
regular expressions to set a search range. 
The search can be further limited by 
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Time Stamp Authority: None 


Help 
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Figure 1: LibreOffice can protect PDF files with a digital certificate or 


encryption key. 


skipping passwords with fewer 
characters than a limit defined with 
--minpw NUMBER (-n NUMBER) or omitting 
any password length greater than the 
number defined with --maxpw=NUMBER 
(-m NUMBER). Yet another way to limit the 
attempt is to use --owner (-o). The initial 
letter case can also be ignored using 
--permutate (-s), an option that is pre- 
sumably intended at some point to 
search for a permutation of a password 
(e.g., search and searching) simultane- 
ously. However, that functionality re- 
mains to be implemented. 


bb@nanday:~/pdfs$ pdfcrack --charset Test 


PDF version 1.6 
Security Handler: 


Standard 


True 


poppler-utils 

The poppler-utils package [3] is a collec- 
tion of several of the most common 
tools for PDF manipulation. When 
checking a file for privacy or security, 
you can use a number of these tools to 
list or extract various objects to be sure 
that you check them: pdfdetach for em- 
bedded files, pdf images for images, 
pdftotext to read the contents easily, 
and pdf info for document extraction. 
However, for security, the most useful 
poppler-utils tool is pdfsig (Figure 3), 
which verifies digital signatures by 


./passworded. pdf 


>: ea@5cd7c2cdb9fd2088695714ccaf65c 
a9ca3f5056baceb/7cd8119db22ef920100000000000000000000000000000000 
0: badad1e86442699427116d3e5d5271bc80a27814 Ff c5e8O0f81l5efeef839354c5f 


found user-password: ‘test’ 





Figure 2: pdfcrack uses a search string to detect a password. 


bb@nanday:~/pdfs$ pdfsig ./blank_signed.pdf 


Digital Signature Info of: 
Signature #1: 


./blLank_signed. pdf 


Signer Certificate Common Name: gerald. holmann@qoppa.com 


Signer full Distinguished Name: 


Signing Time: Sep @1 2016 08:51:37 


Signing Hash Algorithm: 


SHA1 


Signature Type: adbe.pkcs7.detached 


Signed Ranges: 
Total document signed 


[0 - 32976], 


[40532 - 41019] 


Signature Validation: Signature is Valid. 
Certificate Validation: Certificate has Expired 


Figure 3: Part of the poppler-utils, pdf sig detects digital signatures. 
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searching for the NSS certificate data- 
base in the default Firefox profile, as 
well as searching /etc/pki/nssdb. In ad- 
dition, pdfsig shows each signer’s iden- 
tity plus the time and date of the signa- 
ture. Unfortunately, it is not set up to 
check GPG private and public keys. 


QPDF 

Contrary to the impression you some- 
times get from the Internet, QPDF [4] 
is not primarily intended to crack PDF 
files. More accurately, qpdf, a multipur- 
pose command, converts one type of a 
PDF file into another. Among its func- 
tions are linearization (optimization for 
the web); creation of new files; rotating, 
merging, and splitting files; and selecting 
a range of pages to copy to a new file. 
Decryption and encryption are only part 
of QPDF’s functions, offering enhanced 
permissions and higher security than 
PDF’s default permissions. The --help 
option displays a help page similar to a 
man page. 

QPDF can be compiled with several 
different options for encryption [5]. The 
command can be run using 
--password=PASSWORD. If a file has more 
than one password, the option must be 
used for each password, unless all are 
the same. Complete decryption is as easy 
as --decrypt, but encryption can be more 
complicated. The simplest way is to copy 
settings from another file with 
--copy-encryption=FILE, which creates a 
new file even if the original’s password 
is not known. More often, though, --en- 
crypt OPTIONS is used. An encrypted file 
can be created with: 


qodf --encrypt KEY-LENGTH 2 
KEY-LENGTH-FLAGS FILE 


qpdf --encrypt USER-PASSWORD 2 
OWNER-PASSWORD KEY-LENGTH 2 
-- ./INPUT.PDF ./OUTPUT. pdf 


E=gerald. holmann@qoppa.com, CN=gerald.holmann@goppa.com 





FEBRUARY 2022 at 





This command creates the output file 
with no output to the screen. KEY-LENGTH 
can be 40, 128, or 256, with each key 
length offering different flags before the 
input file, as described in the --help op- 
tion, as well as extra security. For exam- 
ple, a key length of 40 allows the default 
permissions included in the PDF stan- 
dard (ISO 32000). A key length of 128 in- 
cludes all the options of a key length of 
40, plus permissions for other altera- 
tions, such as permissions for forms or 
the use of AES encryption. To these per- 
missions, a length of 256 adds support 
for V4 and the deprecated RC5 encryp- 
tion. All these options and flags can be 
used with --password-mode, which sets 
how passwords should be read: literally, 
as UTF-8 encoded, hex-encoded, or auto- 
matically as needed. Any of these op- 
tions can be used with QPDF’s other op- 
tions, to create, for example, an en- 
crypted file with rotated pages or only a 
selection of pages. 


pdf-redact-tools 

Redaction is the removal of private or 
sensitive information prior to releasing 
a document to the public. Often used in 
conjunction with the release of official 
government documents, redaction is 
visible in blacked out words or para- 
graphs. In business, redaction may also 


a 
Ey Se 
apsy eas 
ae Aa 


unencrypted. pdf: 
unencrypted. pdf: 
unencrypted. pdf: 
unencrypted. pdf: 


unencrypted. 


PDF Version: 1.3 


(Microsoft Word - Document1) 


(Word) 


MBN) Command Line - PDF Security 


be necessary, most often when an inter- 
nal document is released to the public. 
pdf-redact-tools [6] aids redaction with 
three options. It turns each page of the 
PDF into a PNG file in a new directory, 
an operation that can also be done 
manually by converting a PDF into a 
multi-page TIF file. With --sanitize 
FILE (-s FILE), a script cleans up possi- 
bly sensitive file names, much like a 
bulk file renamer. However, the most 
useful option is --achromatic FILE (-a 
FILE), which converts color files to 
black and white to thwart identifying 
the printer a file is associated with by 
the use of unique printer dots - a prac- 
tice that is little known to the public but 
widely known by civil rights groups and 
privacy experts [7]. This option, too, 
can be done manually in a graphics edi- 
tor such as Gimp. However, having all 
these tools for redaction semi-auto- 
mated and carried out by a single com- 
mand is convenient. If you have trouble 
using pdf-redaction-tools, which is no 
longer maintained, search for other 
Linux redaction tools online. 


pdfresurrect 

The main use of pdfresurrect [8] is for 
viewing a file’s versioning data. Using -u, 
you can view the versioning history, and 
-q returns only the number of versions 


Version 1 -- Object 73 (Unknown) 
Version 1 -- Object 74 (Unknown) 
Version 1 -- Object 75 (Unknown) 
Version 1 -- Object 76 (Unknown) 


(macOS Version 10.14.6 \(Build 18695\) Quartz PDFContext) 


CreationDate: 
ModDate: 


(D:20190918123039200'00' ) 
(D:20190918123039Z00'00' ) 





Figure 4: Some of the versioning data shown by pdfresurrect. 
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that have been made. With -i, you can 
view the objects in the file, as well as de- 
tails of how the file was created and who 
created the file, if available (Figure 4). 
Any of this information could be sensitive 
— for instance, the versioning could be 
used as proof that the file was changed, 
or the file creator could be used to assign 
blame for the content. To avoid such 
cases, you can use -s to scrub or redact 
all this information. 


Modernizing PDFs 

The PDF format, a relic from a more 
trusting age, is three decades out of date. 
While probably millions use its default 
permissions, these permissions are no 
longer adequate for more than the light- 
est of privacy and security. Today, the 
necessary encryption level has increased 
dramatically, and redaction was not even 
considered in the PDF standard. If pri- 
vacy or security matters to you, I suggest 
that you use one or more of these tools 
with your PDFs. To do otherwise is to 
labor under a false sense of security. BES 


Info 
[1] PDF insecurity: httops:/www.locklizard. 
com/password-protect-pdf/ 


[2] pdfcrack: 
https://sourceforge.net/projects/ 
pdfcrack/files/odfcrack/pdfcrack-0. 19/ 


[3] poppler-utils: https:/okgs.org/ 
download/poppler-utils 

[4] OPDF: https:/gqpdf.sourceforge.io/ 

[5] OPDF options: 


https:/gqpdf.sourceforge.io/files/gpdf- 
manual. html#ref. basic-options 


[6] pdf-redact-tools: https:/github.com/ 
firstlookmedia/pdf-redact-tools 


[7] Printer dots: https:/www.bbc.com/ 
future/article/20 170607-why-printers- 
add-secret-tracking-dots 


[8] pdfresurrect: https:/github.com/ 
enferex/odfresurrect 
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Charly’s Column — getnews.tech 


The sys admin’s daily grind: getnews.tech 








Instead of websites or newsfeeds, Charly prefers to use getnews.tech at the command line to keep 
up to date with what's happening around the world quickly and in a targeted way. By Charly Kiihnast 


hen I want to keep up with 
' what’s happening in the 
| world, I can check my fa- 


vorite online newspapers or, 


if ’m in a hurry, use a news ticker. But 
what if I don’t want to leave the com- 
mand line to do that? I use getnews. tech 
to fill this gap. 

You can easily query getnews. tech via 
the web using cur]. If you want to host it 
yourself to save data, you can find the 


NDTV News 
Flight 


- NOTV 


Pakistan on Wednesday denied further use of its airspace to a 
- operated by re-branded Low-cost airline Go 
that was inaugurated by Home Minister Amit Shah during his visit to 


Srinagar-Sharjah direct flight 
First - 


Jammu and Kashmir last month. 


Calcalist.co.il - ni: 
Last Price - 


HOt F379 


News 18 


iPhone - News18 


Apple iPhone's latest software update i05 14 has introduced Back Tap feature 
to perform simple tasks using the logo on the back of your phone. 


languages and writing systems. 


- Pak Refuses Airspace Permission For New Srinagar-Sharjah Direct 


ign osnen 56% no-s09 n"“ina | 


- We Bet You Didn't Know This Hidden Feature of Apple Logo on Back of 





‘igure 1: getnews. tech delivers up-to-date news in all 


code and installation instructions on Listi 


GitHub [1]. You also need an API 

key from News API [2], which is 

free for private users, but you do 

need to register. News API serves as the 
actual data source, while getneuws. tech acts 
as the CLI wrapper to operate and sort and 
takes care of the human-readable output. 

If I call 


curl getnews.tech 


without fur- 
ther parame- 
ters, I get the 
latest news 
from all over 
the world 
without fur- 
ther sorting. 
Doing this 
means that I 
have to put up 
with receiving 
news in lan- 
guages I don’t 
understand 
and in writing 


:~$ curl us.getnews.tech/n=2, category=science 


ScienceAlert - Mysterious Shards of Glass Are Strewn Across Miles of Desert, 
And We Finally Know Why - ScienceAlert 


They first came to scientists' 
field of glass fragments, 


attention about a decade ago: A mysterious 
scattered across Chile's Atacama Desert, 


and 


aligned in a vast corridor stretching 75 kilometers long (almost 50 miles). 


SciTechDaily - Ancient Exploding Comet Likely Source of Vast Glassy Rock 
Patches in Chilean Desert - SciTechDaily 


Heat from a comet exploding just above the ground fused the sandy soil into 
patches of glass stretching 75 kilometers, a study led by Brown University 


researchers found. Around 12,000 years ago, 


of the Atacama Desert in Chile w.. 


of messages. 


30 


something scorched a vast swath 





re 2: The news can be limited by language, category, and number 


ig 1: Parameterized Call 


$ curl en.getnews.tech/n=2, category=science 


systems that mean nothing to me - He- 
brew, for example (Figure 1), or Japa- 
nese kanji. I only know the kanji char- 
acters for man and woman, which I 
memorized in order to avoid using the 
wrong bathroom in my favorite sushi 
restaurant. 

Fortunately, getnews. tech offers vari- 
ous approaches to organizing this 
chaos. For example, I can prefix the 
URL with a country code to get news 
only in English (curl en.getneuws. tech) 
or exclusively in dozens of other lan- 
guages. In addition, I can select my 
topics of interest. If I want to see the 
latest sports news, I append the param- 
eter category=sports. Besides sports, 
getnews.tech accepts the business, en- 
tertainment, general, health, science, 
and technology categories. 

Lastly, I can specify the number of 
messages that I want getnews.tech to 
show me at any given time. This is 
done by the n=<number> parameter. To 
see the last two messages of the sci- 
ence category from the English-lan- 
guage news stream (Figure 2), I would 
use the call from Listing 1. 


getnews.tech: 


News API: hittps://ne' 


AuUtNO! 

Charly Kuhnast manages 
Unix systems ina data 
center in the Lower Rhine 
region of Germany. His 
responsibilities include 
ensuring the security and 





availability of firewalls 
and the DMZ. 
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Programming Snapshot - 


Game development with Go 


and the Fyne framework 


Chip Shot 


We all know that the Fyne framework for Go can be used to create 


Game Development 


GUls for the desktop, but you can also write games with it. Mike 
Schilli takes on a classic from the soccer field. By Mike Schilli 


he European soccer champion- 
ship a year ago was quite a flop 
for Germany, with what used to 
be a World Cup-winning squad, 
but one scene from the Czech Republic’s 
match against Scotland still sticks in my 
mind. The Scots goalkeeper had run far 
out of the goal, which Czech player 
Patrik Schick noticed while hovering at 
the halfway line. Schick quickly fired the 
ball into the out-of-bounds goalkeeper’s 
goal with an eye-catching arcing shot. 
Since then, I’ve been trying to replicate 
this feat in my position as striker for the 
amateur team “Beer Fit” in San Fran- 
cisco, though without any success so far. 
This is what prompted me to turn this 
into a video game written in Go for my 
Programming Snapshot column. 

The underlying physics for the chip 
shot [1] in soccer is known as “projectile 
motion,” and it’s described in any good 
undergrad physics book. I happen to 
know this exactly because during my 
electrical engineering studies I sweated 
my way through many an exam in the 
murderous “Technical Mechanics” 
course. And even many, many years 
later, holding a totally yellowed degree 


Author 

Mike Schilli works as a 
software engineer in the 
San Francisco Bay Area, 
California. Each month 
in his column, which has 





been running since 1997, 

he researches practical applications of 
various programming languages. If you 
email him at mschi/lli@perlmeister.com 


certificate in my trembling hands, I only 
needed a short refresher to derive the 
formulas for the ball position as a func- 
tion of the starting point, the angle and 
the velocity of the launch, and the 
elapsed time. 

The trajectory of the soccer ball sailing 
over the head of the hapless goalkeeper in 
a high arc into the net behind is by no 
means the only application of these me- 
chanical principles (Figure 1). The same 
long-established formula also calculates 
the trajectories of ballistic projectiles, from 
cannonballs to short-range missiles. 


First Approximation 

To keep the equations (Figure 2) simple 
for the shot’s trajectory in X/Y coordi- 
nates as a function of elapsed time, the 
in-game implementation only takes into 
account the gravity that brings the ball 
back to earth on its arc trajectory, in ad- 
dition to the launch angle and the ini- 
tial velocity with which the attacker 
kicks the ball into the air. It neglects the 
air drag on the ball in the atmosphere. 
This could be incorporated with differ- 
ent flow models, 





all, it’s all | 


about the concept and 
not 100 percent accuracy [2]. 

Listing 1 [3] molds the math into Go 
code and puts it into the chipShot() func- 
tion, which expects the launch speed, the 
angle in radiant format, and the elapsed 
time in seconds as input parameters. It 
returns the position of the ball on the arc 
path at the given time as X and Y coordi- 
nates. Because the formula for the Y-co- 
ordinate on the trajectory is happy to re- 
turn negative values, but the Earth’s sur- 
face does not allow a soccer ball to go 
underground, line 10 sets the height 
value to zero as soon as the flight parab- 
ola assumes negative values. 

For testing purposes, Listing 2 uses 
the Go standard plotter package plot to 
draw the ball’s flight path, with vari- 
ous initial parameters in an X/Y coor- 
dinate system, and generates a PNG 





but then any pre- : 
vailing headwind 
or tailwind would 
also need to be 
considered, along 
with atmospheric 
conditions such 2° 
as fog or drizzle. 
Therefore, the 
program simply 
assumes that the 
ball is flying in a 
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Figure 1: Classic chip shot: The ball flies over the 





Lead Image © Dmitriy Shironosov, 123RF.com 


he will gladly answer any questions. vacuum -— after keeper's head and drops into the goal. 
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Figure 2: Equation to calculate X/Y 
coordinates on the throwing 
parabola, as a function of the 
elapsed time. © Wikipedia 


file. Figure 3 shows the ball’s trajectory 
after launch with a velocity of 10 meters 
per second (m/s) and at an angle of at- 
tack of 45 degrees. If the striker applies a 
little more force to the kick, and the ball 
takes off at 15m/s at the same angle, it 
correspondingly flies higher into the air 
and also covers a greater distance before 
coming back to earth. The launch angle 
is defined by lines 16 through 19 in List- 
ing 2, respectively, in radians rather than 


Listing 1: physics.go 
Ol package main 
02 import ( 
03 ome. 
O4 ) 


O05 fune chipShot(v floaté64, a floaté4, 
t float64) (float64, floaté64) { 


06 const g = 9.81 

Ci eee eee eiimcos (a) 

CS yy =o cE math Sin@a = 6/2 
09 if y<ofg 

10 y = 0 

ia 


Mh return x, y 


Listing 2: plot.go 

Ol package main 

02) Import =( 

03 "sonum.org/v1/plot" 

O4 “sonum.ore/V1/plot/plotter’ 
O05 UPomunlnore/ vill plot pletiatail 
O06 “Sonam, Ore /vl/plot/ ve 

O07 ie hl 

os ) 

Oo 

IOP runes maaan) 4 

Ti p 7] plot. New() 
i pb, litle, Text = “Projectile Motion 
13 p.X.Label.Text = "X" 
14 p.Y.Label.Text = "Y" 


all err := plotutil.AddLinePoints(p, 


16 "v=10/a=45", shoot(10, math.Pi/4), 
7% ‘T-15/o0-45" , shoot (15. math. bi/4 ). 
18 "7 -N0/ aA=GO", Snootlle, math. b1/ 3), 
19 "v=10/a=30", shoot(10, math.Pi/6), 


20°) 
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degrees, just like the sine and cosine 
functions from the math package in Go 
expect it to. Because 180 degrees corre- 
sponds to the value of pi, the function 
only has to calculate the corresponding 
fractions. This means that 45 degrees be- 
comes a quarter of pi and 30 degrees be- 
comes a sixth of pi. 

For each graph, the shoot() function 
implemented in Listing 2 starting at line 
30 defines 20 time points at 0.25-second 
intervals. It uses chipShot() from Listing 
1 to calculate the X/Y coordinates of the 
current ball position and stores the mea- 
surement points in a plotter. XYs type 
array named pts. The function passes 
this array back to the main program at 
the end of the for loop. The main pro- 
gram then uses the AddLinePoints() 
function to pass the data to the plotter. It 
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Figure 3: The ball’s trajectory 
generated by the code in Listing 2. 


21 bs Dey oanaacn ee aibiloes | 
22 panic(err) 
23 } 


draws four of these datasets into the 
same chart as curves into the coordinate 
system, complete with a legend, until 
Save() in line 24 saves the graphic as an 
eight-by-eight-inch PNG file. 


Gamify It 

The remaining listings in this issue turn 
the physics of ballistic trajectory into a 
desktop game named Chipshot. In Figure 
4, the game is in full swing and the user 
has set a ball launch velocity of 15m/s 
with the top slider and a launch angle of 
45 degrees with the bottom slider. The 
goalkeeper is symbolized by the salmon- 
colored rectangle bottom center and the 
soccer goal by the green rectangle further 
to the right. 

With parameters set by the sliders, the 
user presses the Shoot button in the upper 
left corner with the mouse, and the ball 
flies just over the goalkeeper and rolls 
into the goal with its last ounce of energy. 
But this doesn’t always work. For exam- 
ple, Figure 5 shows an attempt where the 
ball flies over the keeper but then dies on 
the way to the goal because it doesn’t 
have enough kinetic energy and just rolls 
to a stop due to friction after hitting the 
eround. Finally, in Figure 6, the ball 
comes down too soon, and the goal- 
keeper catches it. Game over! 

Simple video games of this kind first 
made their way into amusement arcades 
in the 1980s. They were rolled out in 
giant wooden boxes with built-in screens. 


24 err = p.Save(8*vg.Inch, 8*vg.Inch, "curve.png") 


25 dete eer tates, 
26 panic(err) 

27 } 

28 } 

ao 


30 fune shoot(v float64, a float64) plotter.XYs { 


31 hes = 20 


32 pts := make(plotter.XYs, n) 


oe 1 eS OS, 

34 for i <= fanee pis 4 

35 prs. <) prs [aay —sehapSnet(y. 4. ct) 
36 tee Oe) 

ey } 


38 return pts 
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Chipshot 


Shoot Quit 


i | 


Velocity: 15.00 


Figure 4: In this simulation, the ball flies over the keeper's head into the goal. 


Users fed small change into the coin slots 
for the fun of playing the built-in game 
for a few minutes, using a joystick and 
fire button. Younger readers are rubbing 
their eyes in disbelief and raise the ques- 
tion if these poor people didn’t have Play- 
Station consoles at home! The strategies 
behind arcade games and information on 
creating the software that powered them 
are described in Classic Game Design by 
Franz Lanzinger [4], a pioneer of this 
technology. 

According to Lanzinger, he once discov- 
ered an arcade game with the then-popu- 
lar Crystal Castles [5] video game in an 





amusement park in Santa Cruz, Calif. 
With the help of the secret combination of 
the two fire buttons known to him, he 
found out that the visitors to the amuse- 
ment park had put no less than 100,000 


quarters in the machine during its lifetime. 


Extrapolated to the number of 5,000 ma- 
chines produced at the time, this resulted 
in (quite optimistically estimated) total 
revenues for the game of $100 million. 


From Frame to Frame 

What all of these 2D video games have 
in common is that the computer calcu- 
lates and displays the more or less 


Chipshot 


Shoot ult 
eee, 
Webocity: 15.00 


= 


Angie: 4500" 


66606060060608008066006 | 
eo 





Figure 5: Not kicked hard enough: The ball dies on its way to the goal. 
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smoothly displayed movements several 
times per second in what are known as 
frames. Video games are usually based 
on ready-made engines that provide the 
display functionality. They grant the ap- 
plication access to the game events by 
triggering a callback function for each 
frame, in which the game programmer 
then pushes the game characters for- 
ward or checks whether they have col- 
lided with any obstacles in the game. 

The human brain playfully keeps track 
of a complex situation like this onscreen 
so that we immediately notice whether 
the ball in the video game field is heading 
for the goalkeeper or the goal. A software 
program, on the other hand, is dependent 
on repeatedly testing in each game frame 
whether the ball has actually already hit 
one of the monitored objects. The pro- 
gram can do this amazingly quickly, and 
that’s why it looks like it has pattern rec- 
ognition capabilities similar to those that 
humans have - but of course, this process 
is based on an illusion. 


Great Tennis 

The Chipshot video game presented 
below consists of plain-vanilla Go 
code; the graphics library is the plat- 
form-independent Fyne [6], which I 
covered with a photo sorter in a recent 
column [7]. Figures 4 to 6 show the 
game in action. The user pushes the 
two white sliders top left to set the 
launch speed of the ball between 0 and 
30 and sets the angle of attack to a 
value between 0 and 90 degrees. If you 
then click on the Shoot button in the 
top left corner, the ball starts flying on 
its parabolic path. 

As soon as the ball hits the ground 
again, it rolls for a little while longer - 
into the goal with any luck, incrementing 
the Goals counter at the top by one. How- 
ever, if the ball comes down in front of 
goalkeeper, it gets caught, and the keeper 
will potentially laugh gleefully at foiling 
the attempt. The same is true if the at- 
tacker shoots too hard or doesn’t stick the 
boot in hard enough, which causes the 
ball to fly over the goal or die on its way 
rolling towards the goal. 

If the player scores, the score is 
notched up and the game creates a 
new situation by rearranging goal- 
keeper and goal. If the attacker fails 
and the ball does not go into the goal, 
the player can try the same situation 
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Chipshot 


Shoot Quit 


Angle: 45.00° 





Figure 6: Too short: The keeper fields the ball. Game over! 


again with different controller settings, 
but the game resets the goal counter to 
zero as a penalty. 


Programmed Randomness 
Listing 3 sets up the game. To prevent the 
same initial position always coming up 
after a program restart, line 34 uses rand. 
Seed() to set Go’s internal random number 
generator to a value that, when seeded 
with the nanoseconds of the current time, 
yields fairly widely scattered initial data. 

The user interface (UI) elements used 
are defined by the UI type structure in line 
17, which contains the soccer ball, the 
goal, the goalkeeper, and the text displayed 
above the goal or goalkeeper. The global 
variables in the block starting in line 22 
define the dimensions of the playing field 
and the players. The main() function start- 
ing in line 31 first creates a new Fyne ap- 
plication. It then defines a window and 
sets it to a fixed size. It represents the goal 
and goalkeeper as Fyne rectangles in light 
green and salmon pink colors. 

The itemsXPos() function starting in 
line 97 returns the positions for the goal 
and goalkeeper both at program startup 
time and after mastering a standard situ- 
ation for a new round with different pa- 
rameters. It also ensures that no nonsen- 
sical constellations occur, such as the 
goalkeeper standing behind the goal. 

When the goalkeeper intercepts a ball 
or it hits the goal, text appears above 
these game figures, flashing three times 
to report the event. The associated 


eraphical widgets are defined in line 48 
and line 50, but the placeTextHover () 
function (defined later on in Listing 4) 
uses Hide() to ensure that they are ini- 
tially hidden. It is only later that the 
blink() function causes them to flash 
their text several times as needed. 

The ball is shown as a filled circle; it is 
created and filled with red color in line 
45. The virtual ball starts at X position 8@, 
which is the left edge of the field. Line 
52 packs all these widgets into the pitch 
container play, which line 87 later at- 
taches to the buttons and sliders that the 
user uses to influence what happens in 
the game. The two slider knobs velo and 
angle can be moved with the mouse. 
Thanks to Fyne’s binding interface, the 
slider knobs display the set value with- 
out any delay in the associated label in 
each case - very convenient. 

The Shoot button triggers a shot, using 
the values for initial velocity and the ball 
launch angle specified in the sliders. The 
associated callback function starting in 
line 65 first reads the set controller val- 
ues and then calls the animate() function 
from Listing 4. This draws the ball’s path 
onto the playfield and returns a value of 
true if the ball lands in the goal. If it dies 
on the way to the goal, or is intercepted 
by the goalkeeper, a value of false is re- 
turned. This allows the main program to 
keep score. 

In the case of success, the goal counter 
count is incremented by one, and itemsX- 
Pos() defines a new game situation. The 


Fyne Move() function adjusts the widgets 
for the goal and the goalie to the new 
positions, and the associated text panels 
also move with them. As with all 
changes to UI widgets, a call to Refresh() 
is then needed to bring the adjusted 
playing field onto the screen. 

As is common in graphical applica- 
tions, the main program first defines all 
possible responses to user input and then 
enters the infinite main event loop in line 
90 with ShowAndRun(). If the user at any 
point in time clicks the Quit button, 
os.Exit() heralds the end of the program, 
and the UI folds without a sound. 


Action! 

Now, the animate() function in Listing 4 
defines the action on the playing field 
when the user clicks Shoot. Using the 
initial velocity of the ball (velo) and the 
launch angle in degrees, it draws the tra- 
jectory into the game container and eval- 
uates any collisions of the ball with the 
goalkeeper or the goal based on their 
current positions. 

To do this, line 12 converts the degree 
value of the launch angle from the con- 
troller into radian format. The infinite 
loop starting in line 14 processes the 
video game’s animation by calculating 
individual successive frames at intervals 
of 10ms. While doing so, the call to the 
chipShot() function (from Listing 1) in 
line 16 determines the ball’s position on 
the parabolic path associated with the 
current frame for the now time value as X 
and Y values. This is done 100 times per 
second to ensure a smooth animation 
without hiccups. Lines 26 and 27 refresh 
the ball position for each frame; nothing 
else moves on the field during the flight 
phase. 

When the ball ends its trajectory and 
returns to earth, the physics function 
chipShot() returns a Y value of zero, and 
— as asimplified approximation - the 
variable rollout lets the ball continue 
rolling along the ground for another 20 
frames. In reality, it would bounce back 
into the air and only roll out after a few 
hops depending on the substrate friction, 
but the program ignores this to keep the 
formula simple. 

Any collisions of the ball with the goal 
or the goalkeeper are calculated by the if 
constructs in lines 29 and 36; they check 
whether the current ball position is some- 
where within the geometric coordinates 


Shs) 
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Listing 3: chipshot.go 


O01 package main 


002) amportk ( 


003 


OO4 


005 


O06 


O07 


008 


oes) 


010 


O11 


Giz 


O13 


O14 


O15 


O16 


Oly 


018 


019 


020 


G2 


022 


023 


O24 


025 


026 


027 


028 


Oz9 


032 


033 


O34 


035 


036 


037 


038 


039 


O40 


O41 
O42 


O43 


O44 


O45 


O46 


O47 


O48 


(less) 


O50 


O51 


052 


36 


Hae 

col "golang.org/x/image/colornames" 
"image/color" 

"math/rand" 

Nog" 

"time" 

"fyne.io/fyne/v2" 
"fyne.io/fyne/v2/app" 
"fyne.io/fyne/v2/canvas" 
"fyne.io/fyne/v2/container" 
"fyne.io/fyne/v2/data/binding" 
"fyne.io/fyne/v2/widget" 


type Ul struct { 


ball, goal, goalText, goalie, 


goalieText fyne.CanvasObject 


var ( 


gameWidth = float32(1200) 
gameHeight = float32(800) 
goalWidth = float32(30) 
goalHeight = float32(60) 
minDist = oo 


textHover = float32(50) 


fune main() { 


a := app.New() 

WEL Sa) Ue 

rand.Seed(time.Now().UnixNano() ) 

w := a.NewWindow("Chipshot" ) 
w.Resize(fyne.NewSize(gameWidth, gameHeight )) 
w.SetFixedSize(true) 

goalieDist, goalDist := itemsXPos() 

ui.goalie = canvas.NewRectangle(col.Lightsalmon) 


ui.goalie.Move(fyne.NewPos(goalieDist, 
gameHeight-goalHeight ) ) 


ui.goalie.Resize(fyne.NewSize(goalWidth, goalHeight) ) 
ui.goal = canvas.NewRectangle(col.Lightgreen) 


ui.goal.Move(fyne.NewPos(goalDist, 
gameHeight-goalHeight ) ) 


ui.goal.Resize(fyne.NewSize(goalWidth, goalHeight) ) 
ui.ball = canvas.NewCircle(col.Red) 
ui.ball.Move(fyne.NewPos(0, gameHeight-30) ) 
ui.ball.Resize(fyne.NewSize(15, 30)) 

ui.goalText = canvas.NewText("Goal!!!", col.Red) 
placeTextHover(ui.goalText, ui.goal) 

ui.goalieText = canvas.NewText("Caught it!!!", col.Red) 
placeTextHover(ui.goalieText, ui.goalie) 


play := container.NewWithoutLayout(ui.goal, ui.goalie, 
ui.ball, ui.goalText, ui.goalieText) 


053 


O54 


O55 


O56 


057 


058 


059 


O60 


O61 


062 


063 


O64 


O65 


O66 


O67 


068 


O69 


070 


Or i 


O72 


073 


O74 


O75 


O76 


Ove 


078 


O79 


080 


O81 


082 


083 


O84 


085 


086 


087 


088 


089 


090 


Goll 


Co2 


093 


OSM 


095 


ENS 


Cor 


098 


OS!) 


100 


162 


velo := binding.NewFloat() 
veloSlide := widget.NewSliderWithData(0O, 30, velo) 


formVelo := binding. FloatToStringWithFormat (velo, 
"Velocity: %0.2f") 

veloLabel := widget.NewLabelWithData(formVelo) 
veloSlide.SetValue(15) 

angle := binding.NewFloat() 

angleSlide := widget.NewSliderWithData(0, 90, angle) 


formAngle := binding. FloatToStringWithFormat (angle, 
"Angle: %0.2£") 


angleLabel := widget.NewLabelWithData(formAngle) 
angleSlide.SetValue(45) 


countText := canvas.NewText("Goals: 0", &color.Black) 


count O 


shoot := widget.NewButton("Shoot", func() { 


Vv, _ := velo.Get() 
a, _ := angle.Get() 
success := animate(v, a, ui) 


if success { 
count++ 
goalieDist, goalDist := itemsXPos() 


ui.goalie.Move(fyne.NewPos(goalieDist, 
gameHeight-goalHeight ) ) 


ui.goal.Move(fyne.NewPos(goalDist, 
gameHeight-goalHeight ) ) 


placeTextHover(ui.goalieText, ui.goalie) 
placeTextHover(ui.goalText, ui.goal) 
} else { 
count = 0 
} 
eountlext, Text = imt,Sprints:( "Goals: %dl'. count) 
countText.Refresh() 
)/ scoturn) bald to orice in 
ui.ball.Move(fyne.NewPos(0, gameHeight-30) ) 
sD 
quit := widget.NewButton("Quit", 
fuUMe( 4 OS Exit CO sb) 
buttons := container.NewHBox(shoot, quit, countText) 
con := container.NewVBox(play, buttons, veloSlide, 
veloLabel, angleSlide, angleLabel ) 
w.SetContent (con) 


w.ShowAndRun( ) 


fune randRange(from, to int) float32 { 


return float32(rand.Intn(to-from+1) + from) 


fune itemsXPos() (float32, float32) { 


ail 


randRange(minDist, 2*int(gameWidth)/3) 


d2 randRange(int(d1)+minDist, 
int (gameWidth-goalWidth) ) 


return dil. d2 
i 
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Listing 4: animate.go 


Ol package main 34 } 

02 import ( 35 // goalie? 

03 "math" 36 if pos.X >= ui.goalie.Position().xX && 

O4 "time" ay pos.X <= ui.goalie.Position().X+ui.goalie.Size(). 
05 "fyne.io/fyne/v2" Width && 

06 "£yne. io/fyne/v2/canvas" 38 pos.Y > gameHeight-goalYOff-ui.goalie.Size().Height { 
07 39 go blink(ui. goalieText ) 

08 40 break 

09 func animate(velo float64, angle float64, ui UI) bool { 41 } 

TOeenap == 087 / ame 42 time.Sleep(time.Duration(nap) * time.Millisecond * 
An Gee time.Duration(nap) ) 

12 angle = math.Pi * angle / 180 // radient = a a 

ils: LOMOuUt 20 a ; 

14. for { 45 return false 

15 pos == Ul-ball Position () 46 5 

16 Xx, y := chipShot(velo, angle, float64(now)/100) a 

17 if y == 0 { 48 fune blink(tw fyne.CanvasObject) { 

18 rollout-- 49 nen ga desa ey OFS a: <6 oli ee 

19 ie iaellikewne < (6 4 50 tw. Show() 

20 break Sal canvas. Refresh(tw) 

2] 1 52 time.Sleep(250 * time.Millisecond) 

29 i 58 tw.Hide() 

23 goalYOff := float32(30) 54 canvas .Refresh(tw) 

Ou. pos.X = float32(x) * 20 55 time.Sleep(250 * time.Millisecond) 

25 pos.Y = gameHeight - goalYOff - float32(y)*100 56} 

26 ui.ball.Move(pos) 57 } 

27 canvas.Refresh(ui.ball1) 58 

28 // goal? 59 func placeTextHover(tw, w fyne.CanvasObject) { 
29 if pos.X >= ui.goal.Position().X & 60 textPos := w.Position() 

30 pos.X <= ui.goal.Position().X+ui.goal.Size().Width && Gil text Pos.Y — textles.Yo— texthover 

ail pos.Y > gameHeight-goalYOff-ui.goal.Size().Height { 62 tw.Move(textPos) 

32 go blink(ui. goalText) 63 tw.Hide() 

eo return true 64 } 


of the goal or the goalkeeper. They re- 
port a hit with a flashing text if the ball 
rolls into the goal or enable the goal- 
keeper message if the keeper has 
erabbed it. 

Before the for loop moves on to the 
next round, line 42 sleeps for 10ms and 
adds the nap to the current time in nou. 
Then it moves on to the next frame. 
The flashing display for a goal or a 
save is handled by the blink() function 
starting in line 48. It is called on each 
event as a concurrently running gorou- 
tine using go func from animate() so 
that it does not hold up the display. In- 
stead, it runs in the background while 


Listing 5: Creating a Binary 
$ go mod init chipshot 


$ go mod tidy 


$ go build chipshot.go animate.go physics.go 


$ ./chipshot 


the main program can continue to han- 
dle user input. 

The chipshot binary is created using 
the sequence from Listing 5 in the typi- 
cal Go style, with the compiler first re- 
solving the packages used in the code 
and their dependencies from GitHub. 


Coach's Advice 

A few more tips for aspiring young soccer 
players: If the goalkeeper is standing far 
away from the goal and almost in front of 
the attacker, the only way to get past him 
is to shoot the ball up steeply (about 60 
degrees). In this case, the shot needs to 
be powerful so that the ball does not 
come down on its steep path until 
just in front of the goal so that it 
hopefully rolls into it. In game sit- 
uations where the goalkeeper is 
not too far out of the goal, a shot 
with a 45 degree angle of attack 
and moderate speed often does 


the trick. And as always, it helps to prac- 
tice, practice, practice! BEE 


Info 


[1] Chip shot: https:/en.wikipedia.org/ 
wiki/Chip_(association_football) 


[2] Projectile motion: https:/en.wikipedia. 
org/wiki/Projectile_motion 


[3] Listings for this article: ftp://ftp. 
linux-magazine.com/pub/istings/ 
linux-magazine.com/255/ 


[4] Lanzinger, Franz. Classic Game De- 
sign: From Pong to Pac-Man with 
Unity. Mercury Learning and Informa- 
tion, 2nd Edition, May 2019: https:/ 
www.amazon.com/dp/B07S3ZW1Z8 

[5] Crystal Castles: https://en.wikipedia. 
org/wiki/Crystal_Castles_(video_ 
game) 

[6] Fyne: https://fyne.io 

[7] “Programming Snapshot: Go and 
Fyne" by Mike Schilli, Linux Magazine, 
issue 254, January 2022, p. 40 
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We explore some machine learning techniques with a simple 


missing person app. By Roland Pleger 
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=| . 1: Tom’s smartphone shows 


the signal strength from the 
WLANs in the individual rooms at 
his location. 
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complicated. Satellite navigation doesn’t 
work well through doors and rooftops, and 
even if you could replace the satellite sig- 
nal with equivalent transmissions from lo- 
cally placed beacons or WLAN access 
points, the presence of interior walls and 
furniture muddles up the results of classi- 
cal analytical techniques such as those 
used with GPS. What is more, when 
someone is inside a building, the ques- 
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Figure 2: The possible arrange- 
ment of beacons (green dots) and 
positions (blue crosses) in the 
building where Tom is lost. 
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tion is not so much about “What are his 
coordinates.” What you really want to 
know is “What room is he in?” Such a 
problem is better addressed through the 
tools of machine learning. 

Of course, creating a complete machine 
learning solution to find someone in a 
small house might seem like overkill, but 
this article is intended as an exercise to 
show these machine learning tools and 
techniques in a simple situation - a kind 
of machine learning “Hello, World” appli- 
cation. One could imagine scenarios 
where these techniques could find 
broader utility, such as tracking down an 
executive in a large office complex or 
even finding a lost set of car keys. 

In this example, Tom, the protagonist, 
has lost his way. Fortunately, his smart- 
phone shows the signal strength of seven 
hotspots in his vicinity (Figure 1). Be- 
cause Tom often gets lost, I have mapped 
the four rooms as a precaution (the blue 
crosses in Figure 2), and I have a ma- 
chine learning dataset I can use to train 
a program to find Tom. 

With this data, I can use artificial in- 
telligence and supervised learning to lo- 
cate Tom. Supervised learning tech- 
niques are effective, but they require 
some advance knowledge of the house = 
design. What if I know that Tom is in the 5 
house but do not know the number of = 2 
rooms? In that case, unsupervised learn- 5 
ing can help clarify the situation. I will 
explore unsupervised learning later in 
this article. Finally, a method known as 
semi-supervised learning can help me 
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Listing 1: Prepping the Data 


# read data 


import numpy as np 


import pandas as pd 


ial = 


Machine Learning 


Listing 2: Plotting Signal Strength 


Distribution 
dine = dee [de Vlarce || =— = iar rehen,’ | 
dho= dhvdrop( Parget: j~axis = 1) 


dheplotvatst( bine ie. alphna—On 5) 
dh.plot.kde() 


"https: //archive.ics.uci.edu/ml/machine-learning-databases/00422/wifi_ 





NG @ allt eden mie texote 0 61 2 3 4 +#+5 6 _ = Target 

0 -64 -56 -61 -66 -71 -82 -81 Kitchen 

colnames = [0, 1, 2, 3, 4, 5, 6, 'Target'] 1 -68 -57 -61 -65 -71 -85 -85 Kitchen 

df = pd.read_csv(fn, header = None, names=colnames, comment = "#", sep = '\t') 2 -63 -60 -60 -67 -76 -85 -84 Kitchen 

rooms = 71: 'Kitenen, 2: ‘“Hallway’, 3: Livimeroom, 4: Patio} 3 -61 -60 -68 -62 -77 -90 -80 Kitchen 

df['Target'] = df['Target'].map(rooms) 4 -63 -65 -60 -63 -77 -81 -87 Kitchen 
‘oueuliau(ebe |= 2) 


df.describe() 


find any errors I made when assigning 
the rooms. 

This article shows how to navigate in- 
doors using supervised and unsuper- 
vised machine learning methods based 
on Python tools. My focus will be on the 
machine learning technique, with less 
detail about Python and the libraries 
used in the solution. 

For demonstration purposes, I will be 
using a dataset provided free of charge 
from the University California Irvine (UCI) 
website [1]. The UCI Wireless Indoor Lo- 
calization dataset consists of eight col- 
umns: The signal strengths of the seven 
WLAN hotspots, measured using a smart- 
phone, and the location of the measure- 
ments. After 500 measurements at four 
fixed locations, 2,000 entries are available. 

In my scenario, all coordinates are ini- 
tially unknown, both those of the possi- 
ble positions and those of the transmit- 
ting beacons. The signal attenuation is 
measured. Outdoors, you could infer the 
distance using this measurement. How- 
ever, indoors, every obstacle falsifies the 
estimation. Keep in mind, though, that I 
don’t expect to find Tom’s exact position 
in geographical longitude and latitude - | 
only want to find the right room. 


Explorative Data Analysis 
To get started, I need to prep the data (List- 
ing 1). Fortunately, the UCI dataset has 
been preprocessed: It is balanced and con- 
tains no invalid values and no outliers. 

Listing 1 first imports the Python pan- 
das data analytics library [2], which 
supports data handling. The read _csv() 
function reads local files or, as in List- 
ing 1, resources from the Internet. 








Figure 3: The first four lines in the 
pandas DataFrame. 








0 1 2 3 4 5 6 target 

count 2000.000000 2000.000000 2000.000000 2000.000000 2000.000000 2000.000000 2000.000000 2000.000000 
mean = -52.330500 = -55.623500 =©-54.964000 = -53.566500 -62.640500 -80.985000 -81.726500 2.500000 
std 11.321677 3.417688 5.316186 11.471982 9.105093 6.516672 6.519812 1.118314 
min -/74.000000 -74.000000 -73.000000 -77.000000 -89.000000 ~-97.000000 -98.000000 1.000000 
25% -61.000000 -58.000000 -58.000000 ~-63.000000 ~-69.000000 -86.000000 -87.000000 1.750000 
50% -55.000000 -56.000000 -55.000000 -56.000000 -64.000000 -82.000000 -83.000000 2.500000 
75% -46.000000 -53.000000 -51.000000 ~-46.000000 -56.000000 ~-77.000000 -78.000000 3.250000 
max -10.000000 -45.000000 ~-40.000000 -11.000000 -36.000000 ~-61.000000 -63.000000 4.000000 








Figure 4: The statistical description of the data. 


colnames adds the column names 8@ to 6 
to identify the beacons and Target to 
identify the rooms. 
makes the input values tab-delimited. 
If you are working with pandas for 
the first time, you may trip up over the 
index column (the far left column in 
Figure 3). The index column is gener- 


which the dictionary rooms replaces 
with the room names Kitchen, Hallway, 
Livingroom, and Patio. 

The details of the quantiles 25 to 75 
percent say too little about the data distri- 
bution. Instead, I want a graphical repre- 
sentation. Listing 2 initially restricts the 
dataset to the first room with the query 


Using sep = '\t' 


ated automatically and does not come 


from the data, which is why it lacks its 


own column 
header in the 
printed output 
(Figure 3). The 
describe() func- 
tion generates the 
data shown in 
Figure 4: All col- 
umns are fully 
populated with 
2,000 values 
each. The signal 
strengths range 
from -10dB to 
-98dB. The Target 
column takes four 
discrete values, 


['Target'] == Kitchen 
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Figure 5: Scatter of the signal strengths of the seven 
WLAN hotspots in the kitchen. 
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overfitting of a data- 
set, the data is split. 
The greater part is 
used to train the al- 
gorithm. Testing is 
done at the very 
end with the re- 
maining data. The 
prediction’s devia- 
tion from the test 
values is a measure 
of the quality of the 
estimator. 

After reading the 
data as per Listing 1, 
Listing 3 splits the 
DataFrame df into 
properties X and 
target size y. Con- 











Figure 6: The kernel density estimation for all four rooms. 


The drop() command then deletes the 
Target column to remove it from the eval- 
uation. The histogram (Figure 5) is subdi- 
vided into 12 bins. Even with a transpar- 
ency of 8.5, the values overlap. 

The representation becomes clearer 
when the histograms are approximated 
by a continuous function using kernel 
density estimation (KDE) [3], a statisti- 
cal technique used for smoothing proba- 
bility density functions. The bandwidth 
parameter controls the smoothing, but 
adjustment is rarely required. The results 
in Figure 6 are easier to interpret than 
the histogram plot in Figure 5. 

With the data in Figure 6, it now be- 
comes clear that perhaps not the greatest 
care was taken with the measurements. 
Some beacons give partially identical 
signals and therefore no new informa- 
tion about the location. Some curves de- 
viate from a simple distribution. It is 
possible that the data was recorded at 
more than four different positions. The 
high overlap is also one of the reasons 
why an analytical approach will not 
work here. 


Supervised Training 

In each of the four rooms, the signal 
strength of the WLAN hotspots has been 
measured 500 times. Averaging obscures 
too many details to clearly characterize 
the rooms. For example, one of the mea- 
suring smartphones could basically re- 
cord weaker levels, and it would always 
miss the mean value. Machine learning 
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methods come in useful here by putting 
the entirety of the data into context. 

From the wide range of methods for 
classifying, I will limit myself to one 
method known as Random Forest. 

A Random Forest pits many shallow 
decision trees against each other and op- 
timizes internal hyperparameters. A de- 
cision tree starts at the attributes that 
have the highest discriminatory power. 
For example, to separate cherries, plums, 
and apples, querying by size largely di- 
vides the fruit, even if there are large 
cherries and small plums. Next, query- 
ing by roundness of the stone refines the 
subdivision. As the depth increases, the 
prediction improves. In the end, there is 
exactly one query nest for each fruit. 
The problem: The decision tree learns by 
rote. This kind of overfitting is a general 
problem in machine learning. 

Boosting algorithms are some of the 
best classifiers, but an explanation 
would be beyond the scope of this arti- 
cle. Neural networks do not perform 
much better on this problem and are dif- 
ficult to interpret due to the confusing 
number of parameters. 

Machine learning does not perform 
miracles. If the initial data is inconsistent, 
the prediction probability is also limited, 
regardless of the choice of machine learn- 
ing method. The Python libraries scikit- 
learn and sklearn, respectively, support 
all of these procedures. In many cases, it 
is sufficient to replace one line of code to 
classify the data according to a different 


version to a numpy 

array is optional 
and otherwise done later by the classi- 
fier. The train_test_split() function 
splits the data into training and test data. 
In itself, this is a simple task, but the 
routine makes sure that the target vari- 
ables appear equally in both datasets. 
Otherwise, the system might learn Hal1- 
way and Kitchen but not be confronted 
with Livingroom until the testing phase. 

The RandomForestClassifier() com- 
mand selects the Random Forest classi- 
fier. In line 19 of Listing 3, the algorithm 
silently learns its internal parameters. 
The computation time increases with the 
volume of data and parameters. Espe- 
cially for neural networks, it is faster to 
train only once and then store the inter- 
nal parameters (e.g., face recognition in 
OpenCV or cameras works according to 
this method). The parameters learned 
earlier are loaded into memory and rep- 
resent a fully trained system. 

Depending on the method, the inter- 
nal parameters can be several mega- 
bytes. However, small datasets are pro- 
cessed quickly, so I will not elaborate 
on swapping. 


Evaluating the Performance 
If no error message appears after calling 
fit in Listing 3, training was successful. 
However, that alone is not enough. List- 
ing 4 evaluates how well the algorithm 
classified the data. The classifier ob- 
ject contains all the data that has been 
adjusted during training. The classi- 
fier.predict() method calculates target 


Listing 3: Preparing the Classifier 


Ol # split data into features and target 


02 # (change column number) 

03 

OW xX = df, tloe |, 10: -1 |e to munpy() 
OS = dt loc) {= |. tonnunpy 

O6 

O7 # split into training and test data 


08 
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Listing 4: Evaluating the Classifier's Performance 


# predict and evaluate 


from sklearn.metrics import confusion_matrix 


y_pred = classifier.predict(X_test) 


labels = classifier.classes_ 


em = confusion_matrix(y_test, y_pred, labels=labels) 


09 from sklearn.model_selection import train_test_split 


ANG) 


print (np.trace(cem)/y_test.shape [0] ) 


it Xetrain, X test. y train. yotest = train test split(x,. ¥) 


12 
13 # select model and fit 
14 # (change row number) 


aS 


16 from sklearn.ensemble import RandomForestClassifier 


17 


18 classifier = RandomForestClassifier() 


19 Classifier, fit (xX train, yotrain ) 


values for input sequences. I will use 
this tool to find Tom later. To determine 
the quality of the method, I compare 
the test values y_test with y_pred, 
which are the values predicted by pre- 
dict(). The deviations are a measure of 
the quality of the classifier. 

The confusion matrix (Figure 7) com- 
pares the values. It takes the name of the 
target values and their orders from the 
classes_ attribute. The underscore at the 
end of the classes attribute follows a 
common convention of the scikit-learn 
library to mark all values derived from 
the data this way. In Figure 7, the hall- 
way is correctly located 114 times but in- 
correctly identified as the living room 
eight times. Conversely, the living room 
is mistaken for the hallway twice. I will 


pd.DataFrame(cm, 


index=labels, columns=labels) 


Listing 5: Where's Tom? 


pTtom = 


print('Tom is here: 


prant( ‘Error: 


+ OUI put 


a p # Tom is here: 
revisit this prob- 


lem later. 

The confusion 
matrix values are often aggregated to 
create a number: Of 500 values, 12 were 
wrongly assigned, corresponding to an 
accuracy of 1-12/500 = 0.976. Would 
the result be 8.974 if there had been 13 
values? Small numbers have one big 
flaw. The specification of the third deci- 
mal place may be mathematically correct 
in the concrete case, but statistically it is 
wrong. At best, you could limit the error 
to 0.97 + 0.02. 

I have almost found Tom. His 
smartphone shows the field strengths 
of the seven WiFi networks in his en- 
vironment (Figure 1). After training, 
the decision tree classifier derives the 
position from 
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Figure 7: The confusion matrix clarifies the predic- 


tion quality. 
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', classifier.predict([pTom] ) ) 


', classifier.predict_proba( [pTom] )) 


['Livingroom' | 
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Not only does the predict() function 
find the room, predict_proba() also re- 
veals how assured the algorithm is of its 
decision: 77 percent in this case. In Ran- 
domForestClassifier‘s default setting, 
100 decision trees compete against each 
other: 77 decide on the living room, and 
18 decide on the patio. If you were to in- 
crease the number to 1,000 with n_esti- 
mators=1008, you would get values of 
789/1000 and 163/1000. Although this 
technique still leaves some uncertainty, 
it is far better than the analytical alterna- 
tive (see the sidebar entitled “Attempting 
an Analytical Solution”). 


Evaluating the Properties 


this: Tom is in There is a reason why many ensemble 
the living room learning methods rely on decision 
120 a — : 
(Listing 5). trees. Decision trees are robust against 
100 
Listing 6: Prioritizing the Properties 
80 fi = classifier. feature_importances_ 
print('Feature importance: ', fi) 
60 
AO csel = np.where(fi<0.09) 
df.drop(df.columns[csel], axis=1, inplace=True ) 
20 df 
0 # COUN PU: 
# Feature importance: 


7 array ( |O. 25084511, O.00911624." C0, 09055321, 


7 Onel2s2642n O24 906961 OnlO7 S945. O07 (lot om |) 
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Figure 8: How multilateration 
works in principle. 


outliers, and they process categorical 
data that does not need to be metri- 
cally related. You do not need to scale 
the data. And, incidentally, they prior- 
itize attributes. Listing 6 finds a rela- 
tive importance of 9.0 percent for the 
second attribute, WLAN 1, and 7.7 
percent for WLAN 6. The query fea- 
ture_importances <@.1 assigns these 
two indices to the variable csel, 
which then reduces the output data by 
just these two columns. Repeating the 
calculations above with the adjusted 
data yields a similar result: Tom is in 


Listing 7: Converting the Euclidean Distance 


def dom2DistanceConverter(rssi, dbO = -20, N = 4): 


RSSI to distance converter 


Input: mesured power RSSI in dBm; dbO power in 1m 
distance; N attenuation exponent 


Output: distance in meters 


fornia.) Dastance —— 0 


# free space path loss: N=2 
# reduced path loss: N>2 
return 10 


((dibO = essa )/ (iG = N)) 


def eucV(p,b): 





((dbo - RSSI)/(10 * N)) 


the living room with a probability of 
89 percent. 

Redundant data does not affect the ac- 
curacy of machine learning training be- 
cause detecting redundancy is part of the 
training. This is different if redundant 
data slows down the learning process or 
feeds in attributes with contradictory 
data. Later, I will cover other methods 
that do not simply delete attributes but 
try to combine them. 


Unsupervised Learning 
Until now, I have assumed that I know 
the location for each measurement. But 
what if I was careless when noting down 
the rooms? Unsupervised learning looks 
for statements that can be derived from 
the data without contradiction. In this 
case, unsupervised learning would group 
similar measurements together and as- 
sume a common origin. However, 
whether Tom’s location is the kitchen or 
the living room remains undetermined. 

Like in Listings 1 and 3 using super- 
vised learning, the data ends up in an 
array (Listing 8). To distinguish the data, 
I use Xu instead of X. The square brackets 
[:,:-1] delete the target size in the last 
column. To com- 
pare the data, I 
later resort to the 
pandas DataFrame 
df from super- 
vised learning. 

In my experi- 
ments here, I am 


restricting myself to the K-Means clas- 
sifier [4]. The letter K expresses the 
similarity to the k-nearest neighbor al- 
gorithm, which searches for the k near- 
est neighbors, where k stands for the 
number. K-Means divides the data into 
k classes and optimizes the number of 
k centroids such that the sum of the 
squared distances of the points to their 
respective centroids remains minimal. 
Although it sounds a bit abstract, this 
can be programmed with just a few 
lines of code thanks to scikit-learn [5] 
(Listing 9). 

Line 1 in Listing 9 imports the classi- 
fier and line 3 sets up the hyperparame- 
ters. The classifier needs to know the 
number of clusters; I will choose 4 for 
now. The other parameters are default 
values. k-means++ helps the software find 
good initial values, which it optimizes in 
max_iter steps. It makes n_init attempts 
and selects the best solution. random_ 
state starts the pseudorandom generator 
at a defined point, which means that 
each iteration of the computations re- 
turns an identical result. 

Line 5 shows the fruits of my labor: 
The trained method uses kmeans. 
predict(Xu) to assign the measurements 
to the clusters (i.e., in the case of four 
clusters, one of the numbers 0, 1, 2, 
or 3). The seven coordinates of the four 
clusters’ focal points are stored in the 
method variable cluster_centers . 

Listing 10 visualizes the result (Fig- 
ure 10). Strictly speaking, Figure 10 is just 


Listing 9: The K-Means Classifier 


O01 from sklearn.cluster import KMeans 


O02 clusters = 4 


03 kmeans = KMeans(n_clusters=clusters, 


init='k-means++', 


max_iter=300, n_init=10, random_state=0) 


O4 kmeans.fit(Xu) 


OS y_pred = kmeans. predict (Xu) 


o6 clusterCenters 


"""Fuclidean distance between two points squared""" 


return (p[0]-b[0]) 2 + (p[1]-b[1]) 2 


Listing 8: Output Data Without a Location 


import numpy as np 

import pandas as pd 

Import MatpLotlipe pyplot as ple 

fn = "“images/wifi_localization.txt" 


#im = "https; //archive.ics.uei,edu/mi/ 


machine-learning-databases/00422/wifi_localization.txt" O7 


A = ip. loadtxt (tay is. — 1 | 
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kmeans.cluster_centers_ 


Listing 10: Visualizing the K-Means Result 


Ol xi x2 a ty 0 


02 colormap = np.array(['purple', 


'green', 'blue', 'orange']) 


03 plt.figure(figsize=(6,4), dpi=120) 


O04 plt.scatter(Xu[:,x1], Xu[:,x2], s= 10, c=colormap[y_pred] ) 


O05 plt.scatter(clusterCenters[:, x1], clusterCenters[:, x2], 


Ss=180, c='red', marker = 


Oe) 


06 for i, p in enumerate(clusterCenters): 


O08 plt.show() 


plt.annotate(£'$\\bf{i}$', Co[xl]+1, p[x2]+3)) 


Attempting an Analytical Solution 


Satellite-based systems such as GPS or Galileo determine the 
position via multilateration, deriving the distance in a linear way 


from the signal propagation time. 


If you know the position of a satellite (right black dot in Figure 8) 
and its distance (the gray sphere surrounding the black dot), 
you can guess your approximate location. The intersection of 
the gray sphere with the blue sphere (which represents the sur- 
face of the Earth) restricts your location to the green circle. 


A second satellite reduces the possible positions to the inter- 
sections between the green and red circles. If a third satellite is 
added, you are no longer limited to the surface of the blue 
sphere. You can measure altitude as an intersection with the 
cap of a third satellite (not shown in Figure 8). A fourth satellite 
is necessary to synchronize the signal propagation times. Re- 
ceiving signals from additional satellites improves positioning 
accuracy. Modern GNSS receivers can process 30 channels 


and more. 


In the scenario involving Tom, | am not measuring transit times but 
rather the WLAN signal strength, which decreases parallel to the 
square of the distance (see Listing 7). The transmission power is 
unknown, but attenuation is a more serious issue. Any obstacle 
weakens the signal and simulates a — location-dependent — varying 
distance. In the conversion, instead of a quadratic attenuation, a 
number larger than 2is assumed, in this case 4. Because of these 
uncertainties, the following considerations are theoretical. 


This example relies on signal strengths from seven transmitter 
beacons. Unlike GPS satellites, these beacon positions are un- 
known. Instead, their signal strengths are available at four dif- 
ferent locations, and these coordinates are also unknown. The 
altitude is not considered in this example, which means that 
each point is determined by its x and y coordinates. Listing 7 
calculates the Euclidean distance to a beacon. 


There are four unknown positions for the site and seven for the bea- 
cons. In addition, | am also trying to estimate the signal strengths of 
the seven beacons for a total of 29 unknowns. At the same time, | 
know the signal strengths and — in this abstract consideration — the 
distances to the seven beacons, for a total of 28 equations. To solve 
the system uniquely, | need at least one equation for each un- 
known. An overdetermined system of equations would be even 
better to compensate for the errors by means of a fit. 


| fix a location by placing it at the origin. | further assume that a 
beacon is located in the y direction and that the x coordinate 
takes a value of zero accordingly. Finally, | set the signal 


a projection of the seven-dimensional 
property space onto a two-dimensional 
drawing plane. The choice of the @ and 4 
columns in line 1 is not entirely accidental: 
They contain the high-priority features 
found during supervised learning. When 
I look at principal component analysis 
(PCA) [6] later, I will discover another - 
also unsupervised - selection method. 

The plt.scatter instruction prints all 
the measured values, selecting the colors 
from the colormap (line 4). The index for 
the color is the y_pred set in Listing 9. 
The focal points are marked as red 
crosses by the second scatter command 
in line 5. 


Unsupervised 
learning divides 
the data into 
groups and 
chooses the as- 
signments ran- 
domly. In Listing 
9, if the initial 
value of the ran- 
dom random state 
were not fixed, the 
groups would get 
different numbers 
each time they 
ran - I'll come 
back to that later. 
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strength of the transmitter for hotspot 0 such that the distances 
between the positions are on the order of meters. 


That leaves 25 unknowns and 27 equations. While | will get a re- 
sult, the result will not be robust because of the small amount of 
information and the large error in the distance estimate. For 
comparison, GPS requires signals from only four satellites, with 
a linear dependence of distance on time and nearly unob- 
structed views to the satellites. 


Figure 9 shows one example of the nonlinear optimization solu- 
tion, which locates all rooms and beacons. For clarity, the signal 
strengths converted to distance are plotted as circles for the liv- 
ing room only. The other 21 distance circles for the remaining 
three rooms are not shown in Figure 9. 


The transmission power of hotspot 0 is fixed, corresponding to a 
radius of five meters in this case. If the values were robust, you 
would have expected a solution like the one shown in Figure 2, 
where all radii intersect in one point. In Figure 9, it takes good will 
to see where the living room should be in an optimal case. 


So much for attempting to determine Tom’s whereabouts 
analytically with the WLAN data. 
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Figure 9: Example of an analytical solution. 











Figure 10: Visualizing the four clusters found by 
K-Means. 
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Listing 11: K-Means Results 


from sklearn.metrics import silhouette_score 


print(f'Input data shape: {Xu.shape}' ) 
print (£ “inertia: 
print(f'Silhouette: 


print(f'New labels: 


clusterCenters = kmeans.cluster_centers_ 


print(£ Center of eravity: 


Listing 12: K-Means Typical Output 


Input data shape: (2000, 7) 


Inertia 24677146 
Silhouette: 0.41023382751730914 
New labels: [0 1 2 3] 


Center of gravity: [[-35.43058824 ...] 


Hidden Spaces 

Listing 11 provides statistical informa- 
tion about the assignment’s quality; 
Listing 12 shows a typical output. The 
output’s inertia says something about a 
cluster’s compactness. The points 
should be grouped as tightly as possible 
around the cluster’s focal point: The 
smaller the value for the same number 
of points, the better. The output’s sil- 
houette takes into account the distance 
to the neighboring clusters. The farther 
away the neighboring clusters are, the 
clearer the delineation of a cluster. The 


{kmeans.inertia_:3.1f£}') 
{silhouette_score(X, kmeans.labels_)}') 


{np.unique(kmeans.labels_)}') 


{clusterCenters}' ) 
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Figure 11: The elbow method 
finds the optimum cluster size. 
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Figure 12: The silhouette method 
for finding the optimum cluster size. 
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Listing 13: Finding the Optimum Cluster Size 


wess = [] 


for i an rangeCe,. 11): 


kmeans = KMeans(n_clusters=i, 


kmeans. fit (Xu) 


init='k-means++', 


max_iter=300, n_init=10, random_state=0) 


wcess.append(kmeans. inertia_) 


piltt pilot (cance (2. Ii mess | 


plt.title('Elbow Method' ) 


plt.xlabel('Number of clusters' ) 


plt.ylabel('WCSS' ) 


plt.show() 


silhouette value 

lies between 1 

(optimal) and -1 

(possibly wrongly 
set cluster focal points). Both values de- 
scribe the tendency in comparison with 
different cluster sizes with identical ini- 
tial data. 

Listing 13 calculates inertias for differ- 
ent clusters and Figure 11 plots the val- 
ues. The optimum result is a small iner- 
tia for the smallest possible cluster size. 
The elbow method looks for the point at 
which the inertia’s steep slope changes 
to a shallow slope. With a little good 
luck, this point will be at a cluster num- 
ber of 4 or S. Listing 14 does a similar 
job for calculating the silhouette; Figure 
12 shows the results. Again, the best val- 
ues are at 4 and S. 

In Figure 13, each cluster group from 
k=3 to k=8 is given its own subplot. 
The mean values are indicated by a 
vertical red line. In addition, the sil- 
houette value of each dataset appears 
as a horizontal bar, sorted by size. The 
more pointed the right end of the bar 
looks, the greater the variation of the 
values and the more nonuniform the 
cluster. 

With five clusters, the maximum values 
are at a uniform level of almost 0.6. The 


second narrow bar suggests that the data 
contains one small cluster in addition to 
the four large ones. After expanding the 
number of clusters to five in Listing 9 
(i.e., from clusters = 4), K-Means identi- 
fies the set of points at the top of Figure 
10 as a separate group (i.e., a fifth room). 

Unsupervised learning finds connec- 
tions that would have been hidden in 
supervised learning. Using this method 
puts forth the suggestion that the data 
was recorded in five different rooms, 
not four. 


Data Cleanup 

If there is a clear relationship between 
the properties and the target variable, 
the data is redundant. Unsupervised 
learning finds out if the redundancy is 
broken (e.g., due to a write error when 
acquiring the data). 

Line 1 in Listing 15 transfers the input 
data to a pandas DataFrame, and line 2 
assigns the data to one of the four clus- 
ters. The anonymous values @ through 3 
correspond to the rooms encountered 
during the supervised learning example. 
But which four rooms are identified? 

Line 3 in Listing 15 uses the Random 
Forest classifier classifier (trained in 
the supervised learning example) to 
transform the four K-Means cluster 


Listing 14: Plotting the Mean Silhouette Value 


from sklearn.metrics import silhouette_score 


kmeansk = 
inertias = 


Silhouette_scores = 
kmeansk[1: ] | 


plt.figure(figsize=(4,3), dpi=120) 


plt.plot(range(2, 10), silhouette_scores, 


plt.xlabel("Number of clusters" ) 


[silhouette_score(Xu, model.labels_) 


[KMeans(n_clusters=k, random_state=2).fit(Xu) for k in range(1, 10)] 


[model.inertia_ for model in kmeansk] 


for model in 


"bo-") 


plt.ylabel("Silhouette score", fontsize=12) 


plt.show() 
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Listing 15: Data Cleanup Listing 16: Room Assignments Source Data 


Oi din) = pe. Watabrane xu, columns — 0, 2. a. 2.) 5.) 6) cru Cre copy |) 


02 dfu['Target'] = kmeans. predict (Xu) 


03 kList = classifier.predict(clusterCenters ) 
04 transD = {i: el for i, el in enumerate(kList)} 


05 dfiul Target] = drul|"Tarcset” |). map( trans) ) 


focal points back to the target values 
from supervised learning: the identifi- 
ers for the four rooms. Line 4 prepares 
a dictionary for translation using map in 
line 5 to replace the numbers with 
room names. 

Now I can compare the values: Does 
the mapping of the rooms from the 
source data match the clusters that K- 
Means found? To do this, I add an addi- 
tional column Targetu to the DataFrame 
object in Listing 16. The new DataFrame 
object dfgroup takes only the values that 
differ in the target columns. Line 5 
counts the differences. 

Listing 17 shows the output from List- 
ing 16. K-Means recognizes that the liv- 
ing room is a better fit than the hallway 
in 75 cases and better than the kitchen 
in four cases. I already found in super- 
vised learning that the hallway was in- 
terpreted as the living room eight times. 


dfDu['Targetu']= dful['Target' ] 


Further suggestions to clean up the 
data are only hinted at here. In my ex- 
ample, errors in the assignments only 
occur in neighboring positions. It is 
particularly difficult to distinguish be- 
tween the hallway and the living room 
and, to a lesser extent, between the 
living room and the patio. Little 
points to errors caused by careless- 
ness (i.e., completely wrongly as- 
signed rooms). 

In addition, you could consider the 
decision-making statistics from super- 
vised learning and remove the unclear 
values. This improves the classifier’s 
learning ability. For the evaluation, you 
would define a threshold to create 
more categories. For example, the data 
is then classified as “probably living 
room or hallway” rather than a sup- 
posedly unambiguous but actually un- 
certain statement. 





k=3, silhouette=0.355 
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k=7, silhouette=0.306 
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Reducing the 
Dimensions 
Two-dimensional 
diagrams show the 
dependence of two 
parameters, and 
three parameters 
span a three-di- 
mensional space. 
The fourth dimen- 
sion is often illus- 
trated by a time 
stamp on consecu- 
tive diagrams. In 
looking for Tom, I 
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Figure 13: The silhouette values of the individual 


datasets, sorted by size. 


plt.show() 


dfDu[dfDu[ 'Target' ] 
drercup = adipuldtvu|*Parcet” | 


dfgroup. groupby(['Target', 


x = list(range(1,8)) 


f= dfibu | Tarcsetu” || .aloe|?,-27 | 


!= dfDu ['Targetu']].iloc[:,-2:] 


'Targetu'])['Targetu'].count() 


Listing 17: Room Assignments Output 


Target Targetu 

Hallway Living_room res 

Kitchen Living room 4 

Patio Kitchen 2 
Living room 2 

Living room Kitchen 2 
Patio 6 


am dealing with seven component val- 
ues. In order to be able to show the de- 
pendency on a target value, I picked out 
two component values earlier. 

I do this cautiously. From supervised 
learning, I know the components with 
the highest prioritization. PCA con- 
denses the feature’s information and re- 
duces the dimensions without knowing 
the target values. It is a powerful ap- 
proach that also detects outliers. I have 
limited myself to a few use cases. 

Listing 18 turns out to be largely self- 
explanatory. After importing the PCA li- 
brary, the code reduces the number of 
components, in this case from seven to 
seven - so nothing is gained initially. 
However, the method returns the ex- 
plained variance ratio_ attribute, and 
the cumsum function returns the cumula- 
tive sum. Figure 14 shows that already a 
single component returns 65 percent of 
the results correctly, and two compo- 
nents even return 85 percent. 

Given clusters = 4 (i.e., four clus- 
ters), Listing 19 outputs a Voronoi dia- 
gram in Figure 15, which generates a 
cluster distribution similar to Figure 10 


Listing 18: Principle Component Analyis 
from sklearn.decomposition import PCA 


pea_7 = PCA(n_components=7 ) 


plt.plot(x, np.cumsum(pca_7.explained_variance_ratio_ * 


plt.xlabel('Number of components' ) 


plt.ylabel('Explained variance' ) 
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diagram looks 
quite convincing, 
but it doesn’t tell 
me where Tom is 
located. 


Conclusions 
It is impossible to 
calculate Tom’s lo- 
cation using an 
analytical solu- 
tion, mainly due 
to indoor obsta- 
cles weakening 
the signal and pro- 


6 7 








Figure 14: Component meanings. 


without knowledge of the prioritized 
components. The axes have lost their 
old meaning. In my example, seven sig- 
nal strengths give rise to two new scales 
with new metrics. The slightly distorted 
point clouds are mirror images. 

While decision trees do not require 
metrics, K-Means and PCA compare dis- 
tances between datapoints. Typically, 
preprocessing raises the scales to a com- 
parable level. The error by omission fac- 
tor remains relatively small at this point 
because the signal strengths of all attri- 
butes are of a similar magnitude. 

Figure 16 illustrates that preprocess- 
ing plays an important role in estimat- 
ing the number of clusters. Changing 
just one variable, clusters = 18, paints 
a whole new picture. Because of the 
Voronoi cells [7] and the coloring, the 


Listing 19: Voronoi Diagram 


viding varying re- 
sults. To find Tom, 
I instead relied on 
machine learning methods. The methods 
discussed in this article use weak artifi- 
cial intelligence. So far, I have not seen 
any approaches from artificial intelli- 
gence (i.e., self-reflecting systems). 

With supervised machine learning, I 
used the Random Forest classifier to 
categorize new data. K-Means, as an ex- 
ample of unsupervised learning, let me 
look at the data without a target vari- 
able, find interconnections, and evalu- 
ate the quality of the data. Combining 
the Random Forest classifier and K- 
Means, I cleaned up the data using 
semi-supervised learning. 

In addition, using Python’s sctkit-learn li- 
braries ensures easy access to machine 
learning programming. This gives users 
more time to explore the constraints and 
understand the dependencies of the results. 


from Scipy.spatial import Voronoi, vorone1 plot 2d 


from matplotlib import cm 
al ep ret eG, 

# clusters = 4 

clusters = 16 

Xur = pcea_2_reduced 


kmeansp = KMeans(n_clusters=clusters, 
random_state=0 ) 


kmeansp. fit (Xur) 
y_pred = kmeansp. predict (Xur) 


ccp = kmeansp.cluster_centers_[:,[xl, x2]] 


init='k-means++', max_iter=300, n_init=10, 


fie, ax = pilt, subplots( lesize—( 455) 2 sdipi—120 ) 


vere —| Verengl cep) 


voronoi_pllot_ed(vor, ax = axl, line width 


= i) 


pli seat ter One| cil) Xue ce eS 36 eC -v ered cmap. — pllt eet ucmap -vineic1s.)))) 


pltescatter( cep. Ole cep yell .s-150.. c= rede samarker— 


for i, p in enumerate(ccp): 


xe) 


Olt sannotate(£'S\ \briibs'. sGolOl]+1, piljt2)) 
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In the end, I think Tom is probably in 
the living room - or the hallway. Happy 
hunting! S58 
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Figure 15: Voronoi diagram with 
four clusters. 

















Figure 16: Voronoi plot with 18 
clusters. 


Info 


[1] UCI dataset: 
https://archive.ics.uct.edu/ml/datasets/ 
Wireless+lndoor+Localization 


[2] Pandas library: 
https://pandas.pydata.org/ 


[3] Kernel Density Estimation: 
https://en. wikipedia. org/wiki/Kernel_ 
density_estimation 

[4] K-Means Classifier: https://en. 
wikipedia.org/wiki/K-means_clustering 

[5] scikit-learn: https:/scikit-learn.org 

[6] PCA: https://en. wikipedia.org/wiki/ 
Principal_component_analysis 

[7] Voronoi diagram: https:/en. wikipedia. 
org/wiki/Voronoi_diagram 
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-Bird’s-Eye View 


If you don’t speak fluent Ethernet, it sometimes helps to get a graphical view of what your network 
is doing. Skydive offers visual insights that could reveal complex error patterns. By Markus Stubbig 





picture is worth a thousand wiring but about the data flows between — configuration and statistics to the ana- 

, words, and sometimes, a visual the nodes. Skydive stores this informa- lyzer. The analyzer listens to feedback 

ww, image of your network can save tion in a central location. You can inter- from its agents and stores the input in a 
ES WS you hours of troubleshooting. act with Skydive using a web interface, database. Gradually, the analyzer gets to 
Skydive [1] is an open source network the command line, or an API. know the entire topology and traffic 
analyzer designed to provide a graphical Skydive consists of a central analyzer flows between endpoints. The admin 
representation of the IT components and and many agents (Figure 1). The agents can access the new Skydive instance via 
how they interact. I’m not talking about run on Linux hosts and report network the analyzer’s web interface. 





ing 1: Building Skydive on CentOS 

# yum install go git make protobuf protobuf-c-compiler \ 
nom patch libxml2-devel libvirt-devel libpcap-devel \ 
protobuf-devel 

mkdir $HOME/go 


export GOPATH=$HOME/go 


Client 


| -1 
He 


export PATH=$GOPATH/bin: $PATH 


CLI/ Shell | AP — Web-UI 


mkdir -p $GOPATH/srce/github.com/skydive-project 


+O +O FEOOUKOUH 


git clone https://github.com/skydive-project/skydive.git \ 


$GOPATH/src/github.com/skydive-project/skydive 





He 


ed $GOPATH/srce/github.com/skydive-project/skydive 








# make 


‘igure 1: The Linux agents provide states and metrics 
to the Skydive analyzer. 


# make install 


Photo by Michael Olsen on Unsplash 
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Skydive 


Agents act as “dumb” data forwarders. 
The brain of the analyzer is a NoSQL da- 
tabase, either Elasticsearch or OrientDB. 
Skydive scales horizontally: If an ana- 
lyzer is at capacity with its agents, addi- 
tional analyzers can step in and serve 
additional agents. The analyzers fill the 
same database with flow information 
while keeping their configurations in 
sync using etcd. 


Installation 

Skydive’s Github repository [2] provides 
a precompiled binary that is ready to use 
after download. If you don’t trust this 
convenience, grab a build host with a 
compiler and compile the code yourself 
(Listing 1). 

The result is an executable file that 
works as an analyzer or as an agent de- 
pending on how it is called. Workable ser- 
vice files for Systemd are provided in the 
contrib/ directory of the repository. The 
service retrieves the settings from a con- 
figuration file in YAML format, which you 
will find in etc/ in typical Linux style. 


Getting Started 


The all-in-one scenario, which runs the 
agent and analyzer on the same host, is 
a good choice for getting to know Sky- 


Listing 2: Analyzer Using Elasticsearch 
analyzer: 
iicten: O70 .@70.c0e7 
flow: 
backend: myelasticsearch 
topology: 
backend: myelasticsearch 
auth: 
api: 


backend: mybasic 


storage: 
myelasticsearch: 
driver: elasticsearch 
Neste. W727 3ilees 5129200 


ssl_insecure: true 


etcd: 


embedded: true 


auth: 
mybasic: 
type: basic 


file: /etc/skydive/skydive.htpasswd 
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dive. After you call skydive allinone, the 
two components launch immediately 
and explore the operating system and 
the network adapters. The analyzer’s 
web interface listens on Attp:// 
localhost:8082 and lets you browse the 
new environment without any risks. 

The most basic configuration is noth- 
ing but a short interlude because the an- 
alyzer keeps this data in RAM instead of 
using persistent storage on the file sys- 
tem. After restarting the software, all the 
information and settings will be lost. 
However, Skydive is also capable of stor- 
ing the explored topology and flow infor- 
mation in an Elasticsearch database. 
Skydive does not put too much strain on 
the database server. In simple setups, 
the database can run on the same host 
as the analyzer. In either case, Elastic- 
search needs to accept the analyzer’s 
connection attempts. No further knowl- 
edge of Elasticsearch or NoSQL is 
needed; Skydive uses the database more 
like a dumb data silo. 

In the sample configuration from List- 
ing 2, the analyzer learns of its new stor- 
age facility in a neighboring database 
system. It is also important to authenti- 
cate the web interface because —- by de- 
fault - web access works without a 
login. The example uses the modest ca- 
pabilities of Skydive and stores the user 
accounts in the htpasswd file typical of 
Apache. For demanding environments, 
an upstream reverse proxy implements 
almost arbitrary login scenarios. 

If Firewalld controls network access to 
the server, the instructions in Listing 3 
create an exception for Skydive. With 
this exception, the analyzer is ready for 
updates from its agents and the watchful 
eye of the admin on the web interface. 


Agent 
The Skydive agent works like an in- 
former that listens as a regular Linux 
service on a server, on a virtual machine, 
or in acontainer. The configuration file 
tells the agent which protocols and sys- 
tem areas it needs to monitor and which 
analyzer is responsible for it. 

For its discovery tour, the agent can 
explore a wide variety of topologies: 


Listing 3: Firewalld Configuration 


# firewall-cmd --permanent --add-port=8082/tcp 


# firewall-cmd --reload 


Open vSwitch, Docker, OpenStack, 
Linux containers, Libvirt, but also the 
classic neighborhood protocol LLDP. Lib- 
virt support opens the door for exploring 
major virtualization platforms such as 
KVM, Qemu, Xen, and VMware ESXi. 
From the list of topologies, select the 
ones you want Skydive to explore. 

The configuration file in the repository 
contains all the directives with examples 
and default values. By default, the agent 
uses only Netlink and Netns. The config- 
uration file contains only the specification 
of the analyzer and is rather short: 


analyzers: 


- analyzer.example.net: 8082 


Once the agent has launched, it disap- 
pears into the background and commu- 
nicates with its analyzer. The relation- 
ship is a give-and-take affair. The agent 
receives work orders and provides infor- 
mation such as a host’s profile, IPv4/v6 
addresses, network adapter utilization, 
or the ARP and routing table. 

Granted: The same values can also be 
found using the matching Linux com- 
mands. The big advantage Skydive offers 
is that the details from all the agents are 
available before troubleshooting starts. 
The transmitted work instructions ulti- 
mately come from the operator in front 
of the screen and are: collect flows and 
inject packets. 


Collecting Flows 

Up to this point, Skydive has not done 
too much. The colorful topology graph 
offers a neat overview of the network en- 
vironment, but it does not really help 
with problems. 

If connections between end devices 
cannot be established, the network is al- 
ways the first suspect. Networkers need 
to immediately find the glitch and solve 
the problem. This challenge becomes 
even more complex when the devices in- 
volved belong to different teams. Sky- 
dive can help, recording traffic flows on 
the suspicious hosts and listing them in 
the analyzer. Select the icon of an af- 
fected network adapter in the graphical 
web view and launch the Packet Capture 
function in the 
right pane. Just 
like with Wire- 
shark and Tcp- 
dump, a capture 





filter can pick out targeted packets that 
are relevant to the investigation. 

Without anyone noticing, the agent 
collects the flow information of the de- 
sired network adapter and sends it to 
its higher-level controller, which 
dumps the information in the Elastic- 
search database. A look at the Flows 
column of the analyzer provides the 
list of all inspected IP connections 
(Figure 2). The dataset can be sorted 
or filtered as desired using the flow 
query. If the flow you are looking for 
does not appear in the table, the con- 
nection request did not reach that host, 
and the cause of the error must be 
closer to the source. 


Injecting Packets 

Viewing the flow information provides 
passive insights that do not change the 
network traffic. The second main task of 
Skydive is different: forming new pack- 
ets and injecting them into the network 
via any agent. In this way, you can check 
if the test packet arrives at the intended 
destination. 

Assembling a new packet is a conve- 
nient point-and-click procedure in Sky- 
dive analyzer. In Figure 3, the generated 
packet simulates a ping between two 
terminals. Do not type in the required 
source and destination addresses manu- 
ally; simply click on the respective net- 
work icon in the topology view. 


In addition to ICMP, the web UI also 
supports UDP and TCP packets, in the 
IPv4 and IPv6 flavors. Skydive does not 
offer other headers (such as IPsec) or 
complex constructs. For a penetration 
test, this would be a poor harvest, but 
for troubleshooting, these options are 
quite sufficient. 

Speaking of addresses: The Skydive 
agent sends the packet in exactly the 
way the analyzer tells it to. If the com- 
munication between the two endpoints 
goes through a default gateway, the des- 
tination MAC address in the web inter- 
face should be that of the gateway, not 
the destination system. This quirk does 
not emanate from Skydive but from the 
Ethernet protocol. 

Once the packet starts its journey, it 
can no longer be distinguished from a 
normal packet by the switches and rout- 
ers it passes through, thus helping with 
neutral troubleshooting. 


Extending the Topology 
Admittedly, the Skydive agent will not 
run on any old device. But zero-access 
switches and Windows servers will still 
find a place in the Skydive interface 
because the topology can be ex- 
tended to remove the blind spots. 

If there is a connection between Server 
A and Server B that Skydive has not de- 
tected, then the Topology rules menu item 
comes into play. Topology distinguishes 


Skydive 


between nodes and edges. The terms origi- 
nate from graph theory and refer to the 
nodes of a graph and its edges as connect- 
ing lines between the nodes. Skydive uses 
the term node not only for the network 
nodes, but also for their components. For 
example, the server node has an edge to 
the eth@ node, which denotes the server’s 
own network adapter. 

A new node needs a name and a type 
in Skydive. In this case, an unassigned 
icon appears in the topology view to rep- 
resent the new node. A new edge needs 
the identifiers of the two nodes it will 
connect. Just as with packet capture, the 
planned endpoints of the edge can be se- 
lected by clicking. In addition, each edge 
needs a user-definable type, such as 
Layer 2 for an Ethernet connection. 

Skydive places no limits on the con- 
nections. The edge of eth@ does not need 
to lead to another network adapter but 
can also terminate at a block device. 
Skydive’s flexible topology thus provides 
a basis for documentation and visualiza- 
tion. The command line is better suited 
for mass extensions, as described in the 
next section. 


Listing 4: CLI Queries 
# Skydive client status 
# Skydive client query G 


# skydive client query "G.V(). 
Has('Name', 'sdo1s1')" 
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Figure 2: The Skydive Agent reports traffic information to the analyzer. 
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Command Line 
If you don’t want to use point & click for 


troubleshooting, you can use the com- 
mand line instead. The Skydive client 
communicates with the analyzer and 
presents its results in the console win- 
dow. You don’t need an additional pro- 
gram because the client is integrated 
into the Skydive binary. Whether the 
client can talk to its analyzer can be 
checked by posting a simple status 
query (Listing 4, Line 1). 

If the client and the analyzer are not 
running on the same server, the client 
needs the IP address or host name of its 
counterpart in its command call (use the 
--analyzer option). In case of successful 
contact, the display is filled with infor- 
mation about the connected agents, for- 
matted in the JSON format. 

When accessing the entire topology 
tree (Listing 4, second line), Skydive is 
copious and reports every detail about 
every edge and node. It makes more 
sense to use a targeted query that re- 
turns only what you want to know. 
Skydive uses Gremlin as its query lan- 
guage. An example of a query for a 
specific node is shown in the last line 
of Listing 4. 


A bit of basic knowledge in Gremlin is 
needed to create connecting lines in the 
eraph at the command line. The sub- 
command is not query but edge-rule 
create. Listing 5 creates two nodes, as 
well as a connecting edge between 
them. 

Under the hood, the Skydive client ac- 
cesses the Analyzer API. The program- 
ming interface is a regular REST API 
documented in detail via Swagger [2]. 
Access is not limited to the Skydive cli- 
ent but also works with the usual 
HTTP clients Curl, Wget, and Httpie. 
The search for the node in the graph 
from the previous paragraph is handled 
using Httpie with a Gremlin query 
(Listing 6). 


Security 
By default, Skydive does not use en- 
crypted communication. Working with- 
out encryption might be fine for a small 
lab scenario, but a serious setup cries 
out for more protection. Skydive uses 
X.509 certificates to secure the commu- 
nication between the analyzer and its 
agents. 

Skydive does not offer the pre-shared 
Keys variant, so you’ll need certificates 


and a certificate authority. Generating a 
key pair and a certificate involves exactly 
the same steps as for a web server or 
OpenVPN. The analyzer learns about its 
crypto material from a configuration file 
(Listing 7): 

The Skydive agent receives additional 
lines that name the client certificate. 
Every agent always needs its own cer- 
tificate. However, Skydive does not 
grumble if the agents happen to share a 
certificate. 

Encryption starts as soon as the par- 
ticipants are kitted out with certifi- 
cates, the configuration file points to 
them, and the service is restarted. This 
also changes web access to the ana- 
lyzer from HTTP to HTTPS. The add- 
ons in the next section will now also 
access the analyzer via TLS and check 
the server certificate. 

If the dataset is in an external data- 
base, you should secure access. Elas- 
ticsearch has its own certutil tool that 
takes care of the keys and certificates. 
On top of that, there is username- and 
password-based authentication. On the 
Skydive side, the configuration is ex- 
tended to include the credentials for 
the database (Listing 8). 
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4> 
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Nb. of packets 


Figure 3: Skydive can generate IP packets and send them to desired targets on the network. 
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If several Skydive analyzers need to 
keep their data in sync and use the 
key-value database Etcd for this pur- 
pose, the analyzers need to have the 
same level of security. Etcd supports 
certificates and a user login, but Sky- 
dive only uses TLS encryption. Other 
mechanisms need to replace the miss- 


Listing 5: Creating Nodes and Edges 


# skydive client node-rule create --node-name="RT-1" \ 


--node-type="host" --action="create" 


"Name" 0 ret ; 
IMDESeraptaon' 7% % . 


"Metadata": { 


"Name": "RT-1", 
MTrype = host” 
}, 
"Action" > “create” , 
"Query = 1°", 
"UUID": "f2043100-434b-426f-7edc-0382f15d788b" 


# skydive client node-rule create --node-name="RT-2" \ 


--node-type="host" --action="create" 


"Name" ret : 
DCS Eled pedo m s, 


"Metadata": { 


ing authentication, for example, Ipta- 
bles rules or an upstream reverse 


proxy. 


Connected 
As an open platform, Skydive can in- 
teract with other monitoring systems. 
For example, the Grafana visualization 
solution can tap into 
the collected topology 
of Skydive via an ad- 
ditional data source 
and display it graphi- 
cally on a dashboard. 
Skydive provides the 
code for the data 
source in its Github 
repository [3]. In 
order for Grafana to 
access the desired 
content, the query 
needs to use Gremlin 
syntax. In Figure 4, 
Grafana fetches the 
number of concurrent 
IP connections and 
displays them ina 
time-series graph. 
Skydive offers pl- 
ugins for connecting to 


Skydive 


other monitoring solutions. The list is 
(still) quite manageable; in addition to 
Grafana, the only other options are Pro- 
metheus and Collectd. Using the Pro- 
metheus connector, the Skydive analyzer 
provides metrics that the Prometheus 
server collects and processes. With Col- 
lectd, this works the other way around: 
Collectd provides, and the Skydive agent 
consumes. 

If Skydive does not support the moni- 
toring software you are using, there are 
only two ways to get out of jail: write 
your own plugin or tap into the API with 
Curl/Weet. 


Outlook 


The special feature in Skydive is not the 
colorful icons in the topology view, 
which move in a circle across the screen 
every time you click. The treasure is the 
connection data that the agents collect in 
capture mode and report to the analyzer. 
Skydive can process and analyze this in- 
formation. The analyzer does not do the 
work itself but harnesses other tools for 
this purpose. 

The Skydive Flow Matrix add-on pre- 
pares IP connections generated by 
those hosts on which an agent is run- 


Listing 6: Node Search in the Graph 


http POST https://skydive. analyzer: 8082/api/topology 


GremlinQuery="G.V().Has('Name', 


Weyele ven? 2 


Listing 7: Crypto Configuration 


"Name": "RT-2", 
“Types. “host” 
, 
"Action": "create", 
POUk yn ns, 
"UUID": "a8b59b62-2da7-4532-4ac6-6f£94£C898553" 


CALS ¢ 


ca_cert: /etc/ssl/certs/ca-skydive.crt 


# skydive client edge-rule create \ server_cert: /etc/ssl/certs/analyzer.crt 


--src="G.V().Has('Name', 'RT-1')" \ server_key: /etc/ssl/certs/analyzer.key 
--dst="G.V().Has('Name', 'RT-2')" \ 
# Agents need these two additional lines: 
--relationtype="layer2" \ 
client cert, /etc/ssill/certs/cinentil vent 
--metadata="key=value" = / / / / 
i client_key: /etc/ssl/certs/client1.key 
"Name" 0 we ; 
MDeEseriptaem: : =! . i 4t A " " " . 
E Listing 8: Login Information Configuration 
USO GVO) a has (Name bb=se yao ; ; 
storage: client_cert: /etc/ssl/certs/clientl.crt 
"Dst": "G.V().Has('Name', 'RT-2')", 
client_key: /etc/ssl/certs/clientl1.key 
"Metadata": { 
; myelasticsearch: 
"ERelatwondype’: “layer2”, 
i ee if ssl_insecure: false 
key": "value 
eo auth: 
"UUID": "1a429d13-025f-405c-740a-b4bf24bb2763" username: skydive 


} password: uMr8Fv30bX 
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Figure 4: Grafana can use Skydive as a data source to display graphs. 


ning. The resulting list contains the 
protocol, source, destination address, 
port numbers, and address of the 
server that accepted the connection. If 
you find the comma-separated list too 
boring, you can also admire the data in 
the form of a Graphviz diagram or Cir- 
cos ring graph. 

Another add-on offers less eye candy 
but proves useful for security: Security 
Advisor continuously receives flow in- 
formation from the analyzer and exam- 
ines, filters, modifies, and saves the re- 
sults. The results can be stored on Am- 
azon $3, for example, and analyzed as 
Flow Logs using AWS methods. 


Conclusions 

Just as a skydiver admires the beautiful 
landscape below them, Skydive surveys the 
network from a bird’s-eye perspective. The 
information comes from the Skydive 
agents, which collect data on Linux servers 
and report to a central Skydive analyzer. On 
the analyzer, admins can retrieve informa- 
tion about the network via the web inter- 
face or the command line, examine individ- 
ual data streams, and even inject packets 
they define themselves if necessary. The 
added value of Skydive lies in its holistic 
approach, which displays the known net- 
work components in the form of a graph 
and visualizes interrelationships. Bm= 
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Info 
[1] Skydive: 
https://skydive.network 


[2] Skydive API: 
https:/skydive.network/swagger/ 


[3] Skydive Grafana Datasource: 
https:/github.com/skydive-project/ 
skydive-grafana-datasource 
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Many users wish Ubuntu had a free and easily accessible user-driven package repository like 
Arch’s AUR. Pacstall steps into the gap. By Henry ws 


buntu is one of the most popu- 
lar Linux distros, but it is still 
missing some benefits that 
other Linux communities 
enjoy. For instance, the Ubuntu reposi- 
tories might not have the package (or 
the latest version) you need, which 
means you'll need to hunt around 
GitHub for source code to compile that 





Issues with Snaps and PPAs 


Snaps use mount points to load them- 
selves, which makes it difficult to work 
with debugging commands such as 
lsbl1k. Those mount points also are not 
removed after the snap has been re- 
moved, which is very annoying. Snaps 
also take some time to load, which frus- 
trates many users. PPAs are essentially 
mini-repositories for packages, which 
can help users who may want a couple 
up-to-date packages while still keeping 
the rest of the system in line with Ubun- 
tu’s repositories. Unfortunately, PPA’s 
can easily be abandoned without much 
detection and can lead to dependency 
hell when upgrading. 
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package yourself. The two official alter- 
native methods for getting new versions 
of packages (snaps and PPAs) are also 
lacking in many areas (see the “Issues 
with Snaps and PPAs” box). 

When it comes to finding and install- 
ing new software from a broad and di- 
verse user base, many Ubuntu users look 
longingly to Arch Linux and its popular 
Arch User Repository (AUR). The AUR is 
a large community-driven repository. 
The goal of AUR is to provide users with 
easy access to packages that aren’t pres- 
ent in Arch’s official repositories. The 
package format used with AUR is simple 
enough that a developer or maintainer 
from almost any software project can 
easily create an AUR package for their 
software that anyone can use for quick 
and simple package installation. The 
AUR, which currently contains around 
58,000 packages, has helped Arch be- 
come the distro for users who want the 
greatest software availability. 

Many Ubuntu users have wondered if 
Ubuntu could have something like the 
AUR. Enter Pacstall [1] - a command-line 


package manager that aims to bring the 
software availability and freshness of 
Arch/AUR packages to Ubuntu and 
Debian. Like the AUR, the Pacstall project 
contains an open repository of package 
scripts that anyone can add to, improve, 
and utilize. 

The Pacstall project is just getting 
started, and it doesn’t have anywhere near 
the number of packages available in the 
AUR, but the number is growing as more 
users become aware of the Pacstall project. 


How to Install 
To install Pacstall, simply run the 
command: 


sudo bash -c "$(curl -fsSL 2 
https: / /21t.16/dsADi ||| |) 2 
wget -q https://git.io/JsADh -O -)" 


Then, check if Pacstall has success- 

fully installed by running this com- 

mand, which will print your version 
number: 


pacstall —Vy 





Photo by Andhika Soreng on Unsplash 


Using Pacstall 


To search for a package, run: 
pacstall -S foo 


To install a package with Pacstall, for in- 
stance, neofetch (located in the Pacstall 
user repository [2]), simply run the fol- 
lowing command: 


pacstall -I neofetch 
And to install from a local pacscript: 
pacstall -I1 neofetch 


Now if you want to remove a package, 
the command is simple as: 


pacstall -R neofetch 


which will remove neofetch and run post 
removal hooks (if any are available). 
Every so often, you will want to upgrade 
packages, which is done by running: 


pacstall -Up 


If you want to get more information on a 
package you’ve installed, run 


pacstall -Qi neofetch 


Making a Package 

Like the AUR, Pacstall is designed to 
make it easy for any user to create a 
package. For example, suppose I want to 
make a package for st, the simple termi- 
nal. I start by making a file called 
st.pacscript and filling it with the con- 
tents of Listing 1. 

Then I fill in st for the name variable 
and @.8.4 for the version. To find a 
tarball for st, I go to the project website 
[3] and scroll to the bottom to where it 
says Download. Copy the link to the 
tarball (in my case, https://dl.suckless. 
org/st/st-0.8.4.tar.gz), and fill that link 
into the ur] variable. 

St does not specify Ubuntu depen- 
dencies, but, to save you the hassle, 
the one dependency needed is 
libxi1-dev. Put that into the build de- 
pends variable. This will be installed by 


Apt right before the building process. 
Now you need to give a hash to verify 
the authenticity of the tarball when 
you download. (This step is optional 
but highly encouraged.) Simply run: 


wget -q https://dl.suckless.org/2 
st/st-0.8.4.tar.gz && sha256sum 2 
st-0.8.4.tar. gz 


This command will print out the check- 
sum of the tarball. You only need the 
first string, with no file name included. 
Put that into the hash variable. Fill out 
description with a short and concise de- 
scription of st, and fill in maintainer with 
your hame. You can now remove any 
variables not used. In the prepare and 
build functions, add the command true, 
because st does not need to be built be- 
fore installing. 

In the install function, put: 


sudo make clean 2 


install DESTDIR=$STOWDIR/st 


This looks like a fairly standard make in- 
stall except for the ending. The DESTDIR 
will tell make to build $STOWDIR/st, 
which is a path to /usr/src/pacstall/st. 
This step is needed because Pacstall 
will symlink that directory to the root 
directory later, allowing easy file man- 
agement for each package. Now it’s 
time to test the pacscript. Exit your text 
editor and run this command to install 
from a local file: 


Dacstall il st 


This command will install the script you 
just made onto your system. For infor- 
mation on how to make more compli- 
cated scripts, visit Pacstall’s informative 
guide to making a pacscript [4]. 


Conclusion 

There is a reason why official reposito- 
ries like the Ubuntu repositories take 
longer to approve and publish new pack- 
ages. Community-driven projects like 
Pacstall and the AUR do not offer the 
kind of systematic testing and checking 
provided by the distro-branded equiva- 
lents. This gives a community repository 


Pacstall 


Listing 1: Script Template 
O1 name="" 
O02 version="" 
O03 uril="" 
O04 build_depends="" 
O05 depends="" 
OG "description= 
O07 hash="" 
O08 maintainer="" 
O09 
10 prepare() { 
ie 
ie 
ie) alia) 7, 
14 } 
iS 
16 install() { 


17 } 


the freedom to grow and flourish to re- 
flect the needs of the whole user base. 
However, it does put more responsibility 
on you the user to know what you are 
downloading. Many users are willing to 
take this trade-off to gain the flexibility 
and convenience of a community reposi- 
tory. The good news is the scripts are 
typically quite simple, and it is easy to 
see what they are doing by inspection. 
The Pacstall repository currently lists 
around 209 packages, and the number 
continues to grow. It will be some time 
before Pacstall catches up with the 
sprawling size of the AUR, but the devel- 
opers hope that Pacstall continues to 
grow and becomes more useful for the 
Ubuntu/Debian community. #55 


Info 

[1] Pacstall website: https:/pacstall.dev 

[2] Pacstall user repo: https:/github.com/ 
pacstall/pacstall-programs 

[3] st: https://st.suckless.org 

[4] Create a pacscript: https:/github.com/ 


pacstall/pacstall-programs/blob/ 
master/make-a-pacscript.md 
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You can investigate more neat projects 
by Pete Metcalfe and his daughters at 
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Distributed programming made 


easy with Elixir 


Elixir of Life 


The Elixir programming language on a Raspberry Pi lets you 
create distributed projects in just a few lines of code. 


By Pete Metcalfe 





reating distributed and concur- 
rent applications doesn’t have to 
be difficult. Elixir [1] allows 
hobbyists and new programmers 
to create projects that can work across 
multiple nodes. A general-purpose pro- 
gramming language, Elixer runs on top of 
the Erlang virtual machine (VM) [2], 
which is known for running low-latency, 
distributed, and fault-tolerant systems. 

In this article, I look at three projects 
(Figure 1) that use basic Elixir functions, 
with no custom project setup or imported 















Elixir Projects 


Remote Writes i a 
Multinode 
Requests 














Figure 1: Three Elixir projects. 
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libraries. The first project employs remote 
functions between a PC and a Raspberry 
Pi, the second project uses multinode re- 
quests to get Pi statistics, and the final 
project looks at dynamic sharing of data 
between three nodes. 

These projects require only 10 to 25 
lines of Elixir code, showing that dis- 
tributed projects don’t have to be com- 
plicated. 


Getting Started 

For instructions on how to install Elixer 
on your particular system, refer to the 
Elixir website [3]. The process installs 
the Erlang VM and three new executa- 
bles: iex (interactive Elixir shell), elixir 
(Elixir script runner), and elixirc (Elixir 
compiler). 

A good first example is to use the in- 
teractive Elixir shell (iex) on a Rasp- 
berry PI to write to a general purpose 
input/output (GPIO) pin. To begin, 
open the shell with iex and call the 
Raspberry Pi gpio [4] command-line 
tool from the base Erlang :0s.cmd func- 
tion to write a value of 1 to pin 7; then, 
read it back: 


$ iex 

iex> :os.cmd(:"gpio write 7 1") 
[] 

iex> :os.cmd(:"gpio read 7") 
'a\n! 


LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM 
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Elixer BAU GSeNee 


Elixir calls Erlang 


functions when you Use RPC to Remotely Set Pi GPIO Pins 


place a colon (:) in ; 
front of the Erlang PC Raspberry Pi 3 


function or custom 











variable. 

Remote GPIO sane Siex = jesse 
--name pete@192.168.0.120 Connect --name pi3@192.168.0.105 perks is beter 

Control --cookie pitest --cookie pitest 

The next step is to A o22ee 


control a Pi’s GPIO 
from a different 
node. Atwo-node | }}©= 0%... FE 
network can be con- 
figured by defining 
a username with a 
node address and a 
common cookie be- Figure 2: An Elixir remote function call. 
tween the two 


Node.connect :"pi3@192.168.0.105" 
:rpc.call(:"pi3@192.168.0.105",:0s,:cmd ,[:"gpio write 7 1"]) 








nodes. cedure call (RPC) to run an :os.cmd the RPC request, and for this example, no 
For my setup, I logged in to the statement on the Pi node: Elixir code was required on the Pi node. 
Raspberry Pi iex shell with the name 
pi3@192.168.0.105 and a cookie $ iex --name pete@192.168.0.120 2 User Interface 
named pitest (Figure 2, top). Next, I --cookie pitest Now you will want to create a simple way 
logged in to the PC iex session with iex> Node.connect :"pi3@192.168.0.105" for the user to enter a GPIO pin and value. 
the name pete@192.168.0.120 and the true Flixir tends to be used for back-end appli- 
same cookie, pitest. iex> :rpce.call(:"pi3@192.168.0.105",2 cations, but you also have a number of 
From my PC iex session, I only need :os,:cemd ,[:"gpio write 7 1"]) good web server options from which to 
two lines of Elixir code to write remotely [] choose and an Erlang wx graphical module 
to a Pi GPIO pin (Figure 2, bottom). The (a wxWidgets port). 
first line connects to the Pi Elixir node, It’s important to note that the underlying One user interface approach is to use 
and the second line issues a remote pro- Erlang VM on the Raspberry Pi managed the Elixir I0 module to do text console 
reads and writes. The 10.gets() function 
Listing 1: Zen2gpio.exs gets user input, and I0. puts writes to the 
Bie ae eae console. Variables can be inserted into a 


: th Bf < ; he 
02 # Zen2gpio.exs - Use a Zenity form to set the GPIO pin and value input string with t the_variable 5: 


iex> thepin = 2 
04 defmodule Zen2gpio do 
10.gets("Select the Pi GPIO pin: ") 
05 def show_form (pnode) do 
Select the Pi GPIO pin: 7 


O06 thecmd = "Zzenity --forms --title='Set Pi GPIO Pins' --separator=' "7\n" 
' --add-entry='GPIO Pin (0-26)' --add-entry='New Value (0-1)' " 
iex> 10.puts 2 
O7 pininfo = :os.cmd(:"#{thecmd}") 
"Selected GPIO pin: #{thepin}" 
08 # If data is entered in form, write to GPIO and refresh ; 
Selected GPIO pin: 7 
09 it byte size pininnro. )) > 0 do 
10 ‘npe.call(:"pis@le2. 168.0. 105", :0s.emd ~|2"ep1lo write #ipinanto) | ) | : X:~$ zenity -- forms \ 
--title='Set Pi GPIO Pins’ \ 
at show_form (pnode) --separator=' ' \ 
12 end --add-entry='GPIO Pin (0-26)' \ 
--add-entry='New Value (0-1) ' 
13 end 
14 end 
15 Set Pi GPIO Pins 


16 # Connect to the Pi node 


17 pnode = :"p34@192.168.0.105" | GPIO Fin (0-26) 7 
18 Node.connect pnode | New Value (0-1)}| 1 











Cancel 


20 # Show the dialog 





21 Zen2gpio.show_form(pnode ) 


Figure 3: A Zenity form called 
from Bash. 


of 
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¥ Set Pi GPIO Pins 


Forms dialog 


GPIOPIN(O-26) | 7 


New Value (0-1) | | 





Figure 4: Elixir/Zenity remote GPIO write dialog. 





PC 


# Run script file Pi 


$elixir 
--name pi_ temps@192.168.0.120 
--cookie pitest get_temps.exs 









v Pi Diagnostics + X 
pi3@192.168.0.105 
temp=47.8'C 
pi4@192.168.0.101 

temp=45.2'C 


OK 





CPU Temp 


Connect 
rpe:multicall 


Run Multiple RPC Requests 
Raspberry Pi 3 









# Compile script module 

$ elixirc PI_stats.ex 

# Run iex session 

$iex 
--name pi3@192.168.0.105 
--cookie pitest 









Raspberry Pi 4 


# Compile script module 
$ elixirc PI_stats.ex 
# Run iex session 


$iex 
--name pi4@192.168.0.101 
--cookie pitest 


Figure 5: Remote diagnostic from multiple nodes. 


For simple dialogs, I like to use the Bash 
Zenity [5] command-line package. Zenity 
support a number of different dialog types, 
and it is preloaded on Raspian and most 
Linux distributions. 

The zenity --forms command can be 
configured with a presentation that 
asks for a GPIO pin number and value. 
After the user enters data and presses 
OK, Zenity returns a string of the GPIO 
pin number and pin value (Figure 3). 


Listing 2: Pl_stats.ex 


Oe Hessssssssass 

04 defmodule PI_stats do 
O5 def cpu_temp() do 
O06 "#{Node.self()} 
O7 end 

08 # Add more diagnostics like: 


Oo9 end 
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available RAM, 


The Zen2gpio.exs script in Listing 1 
launches a Zenity form to get user input 
— a pin number and value - that is 
passed to the :rpc.call function to do a 
remote GPIO write. 

For this script, the Zen2gpio module is 
created with the function show_form. Elixir 
does not support a while statement; in- 
stead, loops are implemented by recur- 
sion. In this script, the show_form function 
is called initially to open a dialog. An if 


#{:os.cemd(:"/opt/vc/bin/vegenemd measure_temp")}" 


idle time ... 





statement checks for feedback from the 
dialog; if present, the RPC call is exe- 
cuted, and the show_form function is 
called again. No feedback or a press of 
the Cancel button exits the script. 

The Elixir script runner launches the 
code with the common project cookie 
and a unique username (Figure 4). 


Multinode RPC Requests 


The goal for the next project is to have 
a PC node query Pi nodes for diagnos- 
tic information. This project is a little 
different from the earlier project, in 
that a module is loaded on the Rasp- 
berry Pi to send back custom status 
messages (Figure 5). 

To get the Raspberry PI’s CPU, for ex- 
ample, with a Bash command, use: 


# Bash command to get the 2 
Pi CPU temperature 
$ /opt/vce/bin/vegenemd measure_temp 


temp=42.3'C 


This Bash command can be incorporated 
into a small Elixir script (Listing 2) that 
is loaded and compiled on each of the Pi 
nodes. The PI_stats module in the PI_ 
stats.ex script contains the function 
cpu_temp, which returns a string contain- 
ing the Pi node name and the output 
from the shell command to get the CPU 
temperature. 

To compile Elixir scripts, use the elexirc 
command; then, their modules are avail- 
able to iex shells called from that directory. 
The code to compile and then test the PI_ 
stats module from a Raspberry Pi node is: 


+? Compile an Blaixir script 

$ elixirce PI_stats.ex 

## test the PI_stats.cpu_temp 2 

function locally 

$ iex --name pi3@192.168.0.105 2 
--cookie pitest 

iex> PI_stats.cpu_temp() 

{"'pi3@192.168.0.105 temp=47.8\'C\n'} 


An Erlang :rpc.multicall function can 
be used on the PC node to retrieve the Pi 
CPU temperatures. This function is 
passed the node list, module name, func- 
tion call, and any additional arguments: 


iex> :rpe.multicall( 2 
[:"pi3@192.168.0.105", 2 
:"pi4@192.168.0.101"], 2 


PI_stats, :cpu_temp, []) 


Elixer BAEC SeNee 


Listing 3: get_temps.exs :"pete@192.168.0.120" , 2 
Ge ea ee ee ee eee eee (ipis@192 168 0,105") a 
a pat@lo2 168.0. 101" 
O02 # get_temps.exs - get PI CPU temperatures pie 
iex> :rpe.multicall( 2 


O03 # - show results on Zenity Dialog 
allnodes, :mnesia, :stop, []) 
OT a ; 
iex> :mnesia.create_schema(allnodes) 
Of pinedes == |: “pise@lo2. 160.0. 05.) = pit@lo2s ies 7O.. 101 | 


iex> :rpe.multicall( 2 


O6 Enum.map(pinodes, fn x-> Node.connect x end) allnodes, :mnesia, :start, []) 


07 

08 # Get results from remote PI nodes If aschema already exists, you need to 
09 {result,_badnodes} = :rpe.multicall( pinodes, PI_stats, :cpu_temp, []) delete it with the :mnesia.delete_ 

a schema([node()]) call before a new one 


can be created. 

After creating a shared schema, the 
next step is to add a table (Pi3) of GPIO 
pin values for the Raspberry Pi 3: 


11 # Format the output for a Zenity info dialog 
ie output = Enum. map(Ccesnlt fn x => send) |> Enum. join 


13 :0S.emd(:"zenity --imto —-text=\"# output} \" —-titile="Pi Diacnosties'™) 


{["pi3@192.168.0.105 2 shared schema for all the listed nodes. To iex> :mnesia.create_table(Pi3, 2 
temp=47.2'C\n", 2 create a shared or distributed schema, [attributes: [ :gpio, :value] ]) 
"pi4@192.168.0.101 2 Mnesia needs to be stopped on all nodes; 
temp=43.8'C\n"], []} then, after the schema is created, Mnesia For nodes that are writing to a specific 

is restarted. The :rpc.multicall function table, the table should be defined as 
The get_temps.exs script in Listing 3 is is extremely useful when identical actions both a RAM and disk copy. To do this, 
run on the PC to get the Raspberry Pi need to occur on distributed nodes: log in to that node and enter: 
CPU temperatures and present the data 
ina Zenity dialog. iex> # Create a distributed schema iex> :mnesia.change_table_copy_type(2 
To make the code more flexible, all the iex> allnodes = [ 2 Pi3, node(), :disc_copies) 


Pi nodes are stored in a list (pinodes). 
The Eum.map function iterates over the Pi 
node list and connects to each node. 


$ elixir --name pi temps@192.168.0.120 --cookie pitest get temps.exs 


v Pi Diagnostics as 


; pi3@192.168.0.105 
The results from the RPC multicall wy temp=47.2'C 
. pi4@192.168.0.101 
are a little messy, so the Enum.map and temp=43.3'C 


Enum. join functions format the results 
into one long string that is passed to a 
Zenity info dialog box. 

As in the earlier project, the Elixir script Figure 6: Remote Pi CPU temperatures. 
is run with the common project cookie 
with a unique username (Figure 6). 



















Note that once the PI_stats.ex script demic gare hommes Sins Mnesia Project Overview 
is compiled on the Pi nodes, no other ac- es F 
tion is required; as in the first project, New Value (0-1) | 1 
the RPC request is processed by the un- cancel OK ree 
derlying Erlang VM. Distributed Schema 


Data Sharing Between Nodes Tables 


Set GPIO values Ny 
with RPC 

Elixir offers a number of data storage op- 
tions. For simple multinode data sharing, Read back values 
I found that the Erlang :mnesia package ene Lf 
for the Mnesia database management M Ce 
system to be a good fit. In this last proj- sisi 
ect, I set up a shared schema between 2 
the three nodes (Figure 7); the Pi nodes : 
populate tables with their GPIO pin sta- 9 
tus every two seconds. 

On the PC, I use the first project to te 
write to the GPIO pins, and I create a new 
script to monitor the status of the pins 
within a Mnesia shared table. The : mne- 
sia.create schema function creates a Figure 7: Distributed data sharing with Mnesia. 
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Listing 4: Gpio3write.exs 


Ole ee ee 


O02 # Gpio3write.exs - Write Pi 3 GPIO values into Mnesia every 2 seconds 


03 #--------------- 
04 defmodule Gpio3write do 


05 def do_write do 


06 Setpin—w( 2oSs.cemd(; eplo read #1 (Gi) } | tr =—d \"\n\" ") ) 

O7 Enum.map(0..26, fn x-> :mnesia.dirty_write({Pi3, x, getpin.(x) }) end) 
08 :timer.sleep(2000 ) 

09 do_write() 

10 end 

11 end 


12 # Start Mnesia 


13 :mnesia.start() 


14 # Cycle every 2 seconds and write values 


15 Gpio38write.do_write() 


Listing 5: Getting Records 


exo =? Gat all the snecords, use = 
iex> :mnesia.dirty_match_object({Pi3, 


[ 


ere Ow Ona. 
Tews Oe, 
(eee ea 


] 
iex> # Get only Pi Values that are '1' 
iex> :mnesia.dirty_match_object({Pi3, 
[ 

Ie, 2 


rll 


For large projects in which multiple nodes 
are reading and writing into tables, you 
should use transaction statements. For 
small projects that involve just one node 
writing into a table, you can use “dirty” 
reads and writes (i.e., uncommitted data 
in a database). To write the value 1 for 
pin 4 into the Pi3 table and read the re- 
cord back, use: 


iex> :mnesia.dirty_write({Pi3, 4,1}) 

30K 

iex> pintval = 2 
:mnesia.dirty_read({Pi3, 4}) 

[{Pi3, 4, 1}] 


$ zenity --list \ cy 


--title=Pi3 Table \ 
> --text='Pin Values' \ 
> --extra-button Refresh \ 
> --column=Pin 
é 
R 


Pin Values 


--column=VaLlue \ fo _7] 
a) Pee = 


efresh 2 


Tow aul. 


ei 


add a sort at the end 
|> Enum.sort 
et) Enumesort 


Now that you can make simple writes 
and reads, the next step is to create a 
script that continually populates the Pi3 
table with GPIO pin values. 


Populating a Mnesia Table 
The Elixir programming language has 
some interesting syntax features that allow 
you to write efficient code. Two features 
that will help streamline a table input 
function are anonymous and enumeration 
functions. 

The ampersand 
(&) character cre- M 
ates a short hard ees 


Pi3 Table 


$ elixir --name pete@192.168.0.120 


(anonymous function) that can be cre- 
ated on the fly. The following code 
shows simple and complex examples 
that read a GPIO pin value and remove 
the trailing newline character: 


iex> # A basic example 
iex> sum = &(&1 + &2) 


iex> sum.(2, 3) 


iex> getpin=2 
&(:os.emd(:"gpio read #{(&1)} | 2 
tr -d \"\n\""") ) 
iex> getpin. (7) 
i 


The Enum.map function can implement 
complex for-each loops. These two Elixir 
features together can read 27 Raspberry 
Pi GPIO pins and write data to a Mnesia 
table. The Gpio3write.exs script in List- 
ing 4 writes GPIO values into a Mnesia 
table every two seconds. 

The command 


$ elixir --name pi3@192.168.0.105 2 


--cookie pitest Gpio3write.exs 
starts the script on the Pi node. 


Getting Records from Mnesia 
Unfortunately, Mnesia does not support 
SQL syntax, but it does support some basic 
filters like dirty_match_object (Listing 5). 

A dialog from a zenity --list com- 
mand can display table output in col- 
umns (Figure 8). The last argument in the 
command is the data string, which fills in 
the defined columns. The --extra-button 
option returns the button text when the 
button is pressed. 

The script in Listing 6 (Show_gpio.exs) 
reads the Mnesia results. As in the ear- 
lier example, Enum.map and Enum. join 
functions format the results as one long 
string for a Zenity list dialog (Figure 9). 


--cookie pitest Show gpio.exs 





Pi3 Table 


Data string is 
put in columns 


0 
0 
1 
0 
0 
0 
0 
1 


“a oO WwW & WN = O&O 








Cancel 


Figure 8: A Zenity list in Bash. 
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Cancel 





Refresh 


Figure 9: Elixir script showing Mnesia data. 
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- Show_gpio.exs To test that things are working, use 
the first project (Zen2gpio.exs) to toggle 
GPIO pin values; then, the Show_gpio.exs 
dialog can be refreshed to check the new 
values. 

When the final project is running, you 
have an example of Elixir concurrency. 


04 defmodule Show_gpio do 


O05 def getdata() do 


06 result = :mnesia.dirty_match_object({Pi3, :_ ,:_}) |> Enum.sort The Pi3 node is populating a Mnesia 
O7 # create a presentable string for output table, and it is handling remote GPIO 
08 output = Enum.map(result, fn x -> writes and CPU temperature messages. 
"#felem(x,1)} #{elem(x,2)} " end) |> Enum.join 
09 feedback = :os.cmd(:"zenity --list --title=Pi3_Table --text='Pin Values' Final GComments 
eG AC Ee UE Cee one Elixir with the Erlang VM offers a lot of 


--column=Value #{output}" i i 
SO MIE SNELUS COREE SESS functionality out of the box. The next 


= pee eco sueciye aaewmhe aoc ade steps from here would be to look at mes- 
a storuslenel ) saging between nodes and creating proj- 
12 end ects with imported Elixir libraries. & 

3) end 

14 end 

15 # Start Mnesia [1] Elixir: https:/elixir-lang.org/ 

16 :mnesia.start() [2] Erlang: https:/www.erlang.org/ 

17 # Wait for tables to update [3] Elixir installation: 

lg :timer, sleep(1000) https:/elixir-lang.org/install.html 

19 # Show a Zenity list dialog. [4] GPIO utility: 


http:/wiringpi.com/the-gpio-utility/ 


20 # Refresh button to continue, other buttons to quit 


Zenity: https:/help.gnome.org/users/ 


21 Show_gpio. getdata() zenity/2.32/ 
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An Amiga emulator for the 
Raspberry Pi 400 


Wobaante 


Convert a Raspberry Pi 400 into a retro computer that 
behaves like the popular Amiga 500. By Hans-Georg EBer 





he Commodore Amiga was 
more than a gaming platform; 
its graphical interface, Work- 
bench, was way ahead of its 
time. The Amiga models [1] rank 
among the particularly popular home 
computers of the 1980s and 1990s. The 
A500, A1200, and several other Amigas 
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Figure 1: The Amiga A500 Mini is coming in spring 2022. © Retro Games Ltd. 
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had the motherboard and a floppy drive 
integrated into the keyboard case. More 
professional variants like the A3000 
were built more like a PC and housed 
the mainboard, expansion cards, and 
drives in a desktop or tower case. 

According to Wikipedia [1], the 
Amiga 500 is the best-selling model, 
which is what prompted Retro Games 
Ltd. to launch a replica to be released 
in March 2022, dubbed the A500 Mini 
(Figure 1) [2]. The manufacturer is al- 
ready known for reissues of other 
Commodore computers, such as the 
C64 (in miniature format and full size 
with a working keyboard) and VC20 
(full size only). 

For a good Amiga emulation, you 
don’t have to wait for the A500 Mini. 


Emulators are available for practically all 


platforms (even for the old MS-DOS) 
that bring that Amiga feel to other hard- 
ware with varying levels of setup over- 
head. In the Raspberry Pi world, the Pi 
400 is a very good choice for use as an 
Amiga emulator because, like the origi- 
nal keyboard Amigas, it combines a key- 
board, mainboard, and storage device in 
a single housing. 

The developers of the PiMiga [3] had 
the same thought, prompting them to 


build a Raspberry Pi OS distribution that 


boots directly into the Amiberry emula- 
tor [4]. A Pi 400 with the PiMiga card in 
place needs 35 seconds after powering 
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Lead Image © Lightwise, 123rf.com 


on for the Amiga Workbench interface to 
appear in high resolution (1920x1080) 
(Figure 2). The current (at press time) 
PiMiga v1.5 uses Raspberry Pi OS 10 
with kernel 5.10.17. 


Installation 

PiMiga is available as a torrent file, 
which you can source directly from the 
provider [3], in the small or large vari- 
ant. For PiMiga 1.5 Lite, download the 
pimigalSLite.img archive; the archive 
for PiMiga 1.5MF is pimigal5MF.7z, 
which weighs in at 42GB and gives you 
the 77GB pimigal.5MF. img image file. 

It may take some time to download 
the archive, unpack it, and write the 
image file to a sufficiently large microSD 
card (32GB or 128GB). 

To boot the emulated Amiga, you need 
a BIOS, or Kickstart ROM in Amiga- 
speak. Different versions of these ROMs 
are available for various Amiga comput- 
ers. For the Amiga 1200, PiMiga needs 


PiMiga 1.5 MWe 








Figure 2: The graphical user interface on Amiga computers is known 


as the Workbench. 


have to source elsewhere. This process 
is completely legal because all rights to 
the Commodore and Amiga products 
up to 1993 have been transferred to 
Cloanto [5], and the company still dis- 


the correct ROMs therein. Of most im- 
portance is that you have Kickstart v3.1 
for the Amiga 1200 (kick31a1200.rom) in 
the KICK partition on the microSD card. If 
you use an encrypted version of this 


Kickstart 3.1. 

The developer sets great store by 
users sticking to the correct procedure; 
both images come without the required 
Amiga Kickstart ROMs, which you 


tributes software packages containing 
the ROMs [6]. 

The “Kickstart ROMs” box provides 
tips on how to buy a suitable package 
from the Cloanto store and where to find 


ROM (from the Cloanto packages), the 
rom.key file must be in the same loca- 
tion. 

The PiMiga image has been prepped 
in a smart way. Most of the data is on 


Kickstart ROMs 


If you own an Amiga, you can use the original ROM; otherwise, Unpack the AmigaForever9Plus. zip archive after completing the 
you have to get it some other way. One good option is to buy the 
Amiga Forever Plus Edition [7] for just under $30 (EUR30). Man- 


ufacturer Cloanto is the current rights holder of all Kickstart ver- 


download and mount the ISO image af-dvd. iso. Mount the 


image with a variant of the command: 


mount -o loop af-dvd.iso /mnt/ 
sions and sells them, among other things, as part of this pack- 


The Kickstart ROMs are now in the Amiga Files/Shared/rom 
age. After you confirm payment, you can download directly | “ g / /rom/ 


Toi LS Aor pews wAPicln slves vom Shineran il Sl las aller War folder. You need two files for PiMiga: the encrypted Kickstart 


Windows) or a ZIP archive (Figure 3). ROM ami ga-os-310-a1208.rom and the key file rom. Key. 


a @ QQ Suchen Amiga Forever Essentials for Android [8] is far cheaper ($1.99); 





ure,avangate. =) 
but it is only useful if you own an Android smartphone. There is 


no iOS version. Transferring the file involves a bit of work. The 


Amiga Forever 9 Plus Edition (Download) Download Page =, shows the directory with the ROMs (/storage/emulated/@/ 


Download file(s): Android/data/com.cloanto. amigaforever.essentials/files). The 


AmigaForever9Plus. msi 
You have 50 downloads remaining. 
Link expires on: September 01, 2031. 


filenames of the two required files are identical. 


, Copy the two files to the KICK partition on the SD card and over- 
AmigaForever9Plus zip 


You have 50 downloads remaining. 


write the existing rom. key file. Also rename amiga-o0s-310-a1208.rom 
Link expires on: September 01, 2031. 


at this location to kick31a1200.rom. 


Table 1 lists successfully tested ROMs and their MD5 check- 


sums. PiMiga may also be able to handle ROMs from older ver- 








sions of the Amiga Forever packages. With a little luck, you 





Figure 3: Amiga Forever Plus Edition contains might find software collections like Amiga ClassiX 4 or Retro 


various emulators and Amiga games, as well as the 
Kickstart ROMs. 


ClassiX, as well, which also contain the ROMs. The purchase 


price of some of the older DVDs can be less than $10. 
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Figure 4: Try out Amiga games directly in your browser at the Internet 
Archive. 
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Figure 5: Amiga games can be launched conveniently from the 
WHDLoad iGame front end, which comes with a search function. 
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» 1; Kickstart ROMs 
License 


an ext4 partition on the SD 
card. However, in addition 
to the usual FAT partition, 
which contains the KICK 
bootloader, another FAT par- 
tition is present, which 
means you can copy the 
missing ROMs to the card 
from any operating system, 
even if it doesn’t understand the ext4 
filesystem. 


Retro Gaming 

PiMiga does not come with the Kickstart 
ROMs; however, what you will find in 
the images are several thousand games 
that were often sold at the height of the 
Amiga boom in pretty game boxes in 
stores. The community considers these 
abandonware [9], and numerous web- 
sites offer them for download. In the In- 
ternet Archive [10], you can play more 
than 10,000 Amiga games right in your 
browser (Figure 4) from the Software Li- 
brary: Amiga page. 

PiMiga relies on WHDLoad [11] and its 
iGame front end, which you can run by 
double-clicking the iGame icon at the 
bottom of the screen. In this program, 
you can scroll through the entire list of 
installed games, but it makes more sense 
to enter a search term in the top bar; 
then, iGame only shows you the far 
shorter list of matches. 

Many games have a preview image 
(Figure 5), and double-clicking lets you 
open the WHDLoad start dialog for that 
game. When you get there, you can en- 
able cheats (e.g., infinite lives or invul- 
nerability) before you start the game 
with a click on Start - other games start 
directly after a short delay. 

Regardless of the mechanisms pro- 
vided in the game, it is usually possible 
to press F10 to quit the game currently 
running. You are also told this in the 
WHDLoad dialog, which you see di- 
rectly before the game starts. However, 
if the virtual Amiga crashes, this key 
does not work: You need to press the 
keyboard shortcut Ctrl + Fn + Raspberry 
key + Left arrow to restart the emulator. 


Joystick Setup 

Before you start your first game, you will 
want to connect and set up a joystick or 
gamepad. To begin, press F12 to enter 
the Amiberry configuration. (On the 
Raspberry Pi 400 keyboard, F12 is 
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~/ About 


= Paths Port 8: Mouse 
» Quickstart No autofire 
= Configurations : THEC64 Joystick 
= CPU and FPU 
= Chipset = autof ire 
i ROM Swap ports 
u RAM 
Port 2: (none> 


= Floppy drives 
2 Hard drives/CD 
= RIG board 

@ Display 

* Sound 


No autofire 
Port 3: <none> 
No autofire 
None 


Custon controls "aude Shick © 


Miscellaneous Mouse Stick 1: None 
S Priovity Autofire Rate: Fast 
of Savestates 
Virtual mouse driver 
* Both Native only 
“Shutdown = Quit Restart Help 





+ Default + 
THEC64 Joystick 





~ Joystick 


v Mouse/Joystick autoswitching 
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~ Digital joy-nouse speed: 18 
~ Analog joy-mouse speed: 168 
+ Mouse speed: 168 
Magic Mouse untrap 
Host only 
Reset Resume 








Figure 6: Configure your joystick or gamepad to avoid having to 


control games with the keyboard. 


Fn + F2.) Click Input on the left and then 
select the device on Port 1 in the drop- 
down menu. Directly below, you can 
then change the drop-down from Default 
to Joystick or Gamepad (Figure 6). 

Switch to Configurations on the left 
and click Save to save your changes. 
When you return to the desktop with Re- 
sume, the joystick should work in all 
games. 


Complete System 

The Amiga Workbench is not just a pretty 
game starter. Various applications and 
tools are installed that you can try out. 
AmigaAMP, for example, is an MP3 player 
that works like the classic WinAMP or its 
Linux clone XMMS. 

The version of Workbench used by 
PiMiga is more modern than the offi- 
cial Workbench shipped by Commo- 
dore, and it takes advantage of the 
greater power of current systems. (If 
you want to take a look at older Work- 
bench versions, you will find a live 
simulator online [12].) 

Folders with icons on the desktop can 
be opened by double-clicking. By the 
way, Amiga computers use a different 
nomenclature than the rest of the 
world. The desktop is known as the 
Workbench on the Amiga, and disks 
have hierarchical filesystems, folders 


The default file manager for PiMiga is 
Directory Opus. The program offers a 
two-column view and supports various 
file operations on the selected file(s) 
from buttons at the bottom. In this way, 
you can copy, move, or delete files quite 
comfortably. The operating principle is 
somewhat reminiscent of Norton Com- 
mander (MS-DOS) or its clone Midnight 
Commander (Linux). 

By default, Directory Opus creates a sec- 
ond workbench on which it then runs in 
full-screen mode. You can change this if 


necessary by right-clicking on the me- 
nubar and selecting Configure | Screen | 
Screen mode | Display mode... Workbench: 
Use. You can switch between the two 
screens at any time with Ctrl + Alt + Shift. 
The way program windows are han- 
dled is similar to other graphical user 
interfaces: You can move windows 
from the upper window border and 
change window size by grabbing the 
handle in the lower right corner. Click- 
ing once on a window shifts it into 
focus, but you need two clicks (again 
on the titlebar) to bring it to the front. 
One of the icons on the right edge of 
the titlebar alternately moves the win- 
dow all the way to the front or all the 
way to the back; to close a window, 
single-click in the upper left corner. 


Internet 

If the Raspberry Pi is connected to the 
network over Ethernet, the Amiga pro- 
grams will have access to the Internet. 
Whether or not this is useful depends on 
which services you want to use. For ex- 
ample, the standard AWeb browser in 
the version installed on PiMiga is not 
SSL-capable and therefore does not work 
with sites that no longer provide an un- 
encrypted version. 

The browser can nevertheless be 
useful for some tasks. For example, 
you can access Aminet [13], which of- 
fers a comprehensive software collec- 
tion. As a test, I downloaded an Amiga 
version of the C shell as an LHA ar- 





Workbench Screen 


esh550a. lha 


esh55Ga. lha. info 
datei 

demo. sh 

HISTORY 
testdatei 





Figure 7: You can download additional software from the Aminet 
server with AWeb and then unpack and start the program. 


are known as drawers, and subfolders 
are subdrawers. 
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chive, unpacked it in Directory Opus, 
and finally launched it from the termi- 
nal program (Figure 7). 


Login and SSH 

The Raspberry Pi does not start a clas- 
sic Linux desktop. The Workbench in- 
terface is a fixed part of the emulation 
and cannot run Linux applications. 
However, you can get this to work with 
a trick that lets you install a VNC 
server on the Raspberry Pi. 

First you need a Linux shell. If you are 
running another computer on the net- 
work with a secure shell (SSH) client, 
log in to the Raspberry Pi with 


ssh pi@pimiga4oo 


and the login password pimiga. 

If name resolution is not working, 
you need to find the IP address of the 
Raspberry Pi. The fastest way to do this 
is to press F12 to open the Amiberry 
menu and click Quit at the bottom. You 
will then end up in text mode, where 
you are logged in as the pi user. Enter 
the command: 


$ hostname -I 


192.168.178.118 192.168.178.117 2 


2001:16b8:a43:££00:ed90:df202 
:397b:bfda 2001:16b8:a432 
:££00:bf£02:667c:59db:93c6 


The first address shown in the output 
that follows the command is the IPv4 
address assigned to the network card. 
Another IPv4 address (for WLAN) and 
the associated IPv6 addresses might 
follow. 

If you have already quit the Amiga 
emulator, execute the steps directly in 
the shell on the Raspberry Pi; you can do 
without the remote login from another 
computer in this case. 

To begin, install the required packages: 


sudo apt update 
sudo apt install tightvnceserver 2 


xfce4 xfce4-terminal 


Most important is the TightVNC server 
[14], and you need some kind of win- 
dow manager; the Xfce lightweight desk- 
top works well in this case. 
Alternatively, a simpler window 
manager (without a desktop environ- 
ment) will do the job. The obvious 
candidate is AmMiWM [15], which mim- 
ics the Workbench; unfortunately, it is 
not available as a ready-to-use package 
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for Raspberry Pi OS, and I was not im- 
mediately able to compile it from the 
source code. Working as the pi user, 
the commands 


vneserver 


vnepasswd 


let you assign a VNC password and start 
the server. If you had to quit the Amiga 
emulator for these steps, you can restart 
the emulator now: 


ed; sudo ./amiberry.sh 


On the Amiga workbench, open the Sys- 
tem, Internet and TwinVNC drawers in 
turn; then, double-click to launch the 
TwinVNC program and enter the address 
localhost:5901 for Server and the pass- 
word you just set for Password. Now 
click Connect (or press Enter): TwinVNC 
opens the connection to the locally run- 
ning VNC server. 

Initially, the Xfce desktop prompts you 
to set up the panel (Welcome to the first 
start of the panel). Next, click Use de- 
fault config. After that, you can work 
with the desktop (Figure 8), much like 
any other Linux installation, but with a 
few limitations. Scrolling with the mouse 
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Figure 8: Running a full-blown Linux desktop on the Raspberry Pi 400 next to the Amiga emulator does 
disturb the retro look a bit, but it is practical. 
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does not work in all programs, and the 
middle mouse button is not supported; 
therefore, you have to resort to keyboard 
shortcuts like Ctrl+C and Ctrl + V for 
copy and paste actions or Ctrl + Shift + C 
and Ctrl + Shift + V in the terminal win- 
dow. Alternatively, you can use menu 
items (e.g., Edit | Copy). Copy and paste 
between Linux and Amiga programs did 
not work in our lab. 

On the running Linux desktop, you 
can also start a state-of-the-art browser 
like Firefox, although you need to in- 
Stall it first: 


sudo apt install firefox-esr 


By the way, to grab a screenshot of the 
Workbench, install the Linux raspi2png 
[16] tool. Every time you run the pro- 
gram, it then stores a screenshot in the 
current folder as snapshot.png. This 


PiMiga 2.0 


Just before this issue was to go to the 
printer, PiMiga developer Chris Ed- 
wards gave everyone an early Christ- 
mas present by publishing the brand 
new PiMiga 2.0 release. In a YouTube 
video [17], he shows off new features, 
such as the ability to switch quickly be- 
tween three configurations, one of 
which is optimized for classic Mac OS 
emulation with ShapeShifter. The video 
description also contains the download 
link for PiMiga 2.0. 


command also works on another com- 
puter if you are logged in over SSH. 


Add 100 
Thanks to PiMiga, the Raspberry Pi 400 
morphs into the Amiga 500 - or a succes- 
sor to the classic home computer - with 
state-of-the-art ease of use and reasonable 
performance. Much of what PiMiga can 
do could be built as a DIY project (e.g., 
with DietPi [18]) in the Amiberry variant; 
however, that would mean a huge 
amount of work and deep diving into the 
configuration of various tools. 

PiMiga impresses as a ready-made, 
complete package that only lacks the 
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https:/www.amigaforever.com/ 
android/ 
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Kickstart ROMs. The microSD card is 
filled to the brim with classic games, 
so you can have weeks or months of 
fun trying out these old treasures. Even 
better, you can play with version 2 
now (see the "PiMiga 2.0" box). Have 
fun diving into the Amiga software 
universe! HEE 
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INTRODUCTION /LINUXVOICE 


Presentations are one of the items most affected by the 
spell of past Microsoft dominance. Lots of people in the 
business world dont even use the word ‘presentation’ and 
refer to their slide deck as a “PowerPoint” — as if that were 
a real word in the language. The open source world can 
rival Microsoft's clunky and exclusionary PowerPoint with 
Impress, a versatile tool that is part of both the OpenOffice 
and LibreOffice suites. But if you're looking for something 
beyond the usual lines, text boxes, 

and clip art, youmight wantto plan aw == 
your presentation around a full- 
featured video editor like 
Kdenlive. This month we show 
you how! Also inside, we 
compare some leading file 
exchange tools and delve 
into Prettymaps, a 

Python package for 
drawing maps with 
OpenStreetMap data. 
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DOGHOUSE — NEW/OLD COMPUTERS 


MADDOGS 
DOGHOUSE 


Computer architectures from the 1960s and 1970s are 
given new life via modern kits. sy son “mappoc” HALL 


Reviving old computers 


on TV) was one that used punched cards. It had a fairly 
primitive “Disk Monitor System’ that would allow storage 
of programs and data on a very small (by today’s standards) disk. 
| do not remember much about the machine because com- 
puters were really not my interest at the time. | was studying to 
be an electrical engineer, and during the cooperative education 
part of my program at Drexel University (née Institute of Tech- 
nology) | took a course in “How to Program the IBM 1130 in 
FORTRAN’ just for fun. And it was fun. But so were electronics. 
| went back to Drexel and found another computer in the elec- 
trical engineering labs. It was a Digital Equipment Corporation 
(DEC) PDP-8 computer, with 4,096 12-bit words of core mem- 
ory. This computer was programmed through a terminal, the 
ASR-33 Teletype with a paper tape reader and punch. This 
was the machine | fell in love with, and | spent most of my 
waking hours (when | was not flunking out of electrical engi- 
neering [1]) programming it in assembly language. 

The PDP-8 was so “easy” to program in assembly language 
because it had only eight basic instructions, and every one of 
those instructions was exactly 12 bits long. All of the addresses 
were also 12 bits long, as the PDP-8 was a “word” machine. We 
would read a 12-bit word from memory, and then shift it ina 
register to access the pieces we wanted. 

| learned to program in assembly by reading a paperback 
book published by DEC and given to me by a salesman from 
DEC. | think the book probably cost about $5 in those days, but 
that was still a lot of money for a college student because it 
could otherwise buy 10 pitchers of beer. 

After university my assembly language experience (and 
knowing how the actual hardware of the computer worked) 
would help me understand more and more about computers 
and help me get the jobs | wanted. 

lam mentioning all of this as a backdrop to my recent pur- 
chase of some interesting old/new computers. Old because 
their architectures came from the 1960s and 1970s, and new 
because the machines that used to cost thousands of dollars 
and take up a room (with associated air conditioning and three- 
phase power needs) now can fit into an Altoids tin and run off 
AA batteries or power from a USB port. Some can utilize emula- 
tors and simulators on your GNU/Linux machines. 

To go into great detail on all of them would be hard to do in 
this article, but | can give you pointers for the ones | find inter- 
esting, and you can go from there. 


T he first computer | programmed (or saw in real life and not 
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PIDP computers simulate DEC PDP-8 (PiDP-8) [2] and PDP-11 
(PiDP-11) [3] computers, complete with the front panels of blinky 
lights and toggle switches, which allow you to stop the computer in 
mid-stride and examine and deposit data in various parts of “real 
memory. The project web pages point to the various software writ- 
ten for them decades ago as well as additional programs written 
more recently by die-hard fans who love programming for ma- 
chines whose memories were measured in kilobytes rather than 
megabytes or gigabytes. They come in kit form so you have to be 
good at soldering, but | was able to solder the PIDP-11 together 
and have it work on the first try. 

One of the interesting things about these kits is that they use a 
Raspberry Pi running a program called SIMH to host the other 
code. SIMH is free software that can simulate old computer sys- 
tems such as the PDP-8 and PDP-11, as well as control the con- 
sole lights and switches. It was started by an old friend of mine 
named Bob Supnik who used to be a vice president for DEC. SIMH 
is now maintained by a series of volunteers. 

SIMH also has a /ot of software for it [4]: old DEC operating 
systems and compilers (unfortunately only in binary), as well as 
early Unix systems — some with source code available. 

Two other new/old systems are the 1802 Membership 
Card [5], which recreates the COSMAC ELF (RCA 1802) and 
the Z80 Membership Card [6]. 

These projects by Lee Hart allow you to investigate two very in- 
teresting microprocessors from the 1970s and an operating sys- 
tem called CP/M that ran on hundreds of thousands of systems at 
a time when Bill Gates could not even say the word “BASIC’ [7]. 

The Z80 Membership Card even allows you to put massive 
(for that day) “disks” onto a single microSD card supplying the 
programs and data through your own laptop or desktop. Hmm 


1] It was not exactly electrical engineering that | was flunking. It 
was Fourier Analysis and LaPlace Transforms. 





[2] PiDP-8: https://obsolescence.wixsite.com/obsolescence/pidp-8 
[3] _PiDP-11: https:/obsolescence.wixsite.com/obsolescence/pidp-11 
[4] SIMH software: http://simh.trailing-edge.com/software.html 
[5] 1802 Membership Card: http://www.sunrise-ev.com/1802.htm 
[6] _Z80 Membership Card: http:/www.sunrise-ev.com/z80.htm 
'7| This is not true. Bill could say the word “BASIC.” 
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SLIDESHOWS WITH KDENLIVE 





Kdenlive plays to its strengths when editing larger video projects and also helps 
users create appealing slideshows with impressive effects. 


ranted, at first glance, it seems like total 

overkill to create a slideshow with a video 

editing program. But if you take a closer 
look, there are good reasons to do so. 

It is true that even simple programs such as 
Imagination [1] let you create slideshows relatively 
quickly from a series of images. Many transitions 
are available, and zooming is pretty easy. But when 
it comes to fine-tuning the zoom speed or holding 
Zoom points and then moving the section further, 
this kind of software quickly reaches its limits. 

This is where video editing programs come into 
their own. Although the transitions they offer are 
only suitable for slideshows to a limited extent, they 
can be used to control zooms and pans in all varia- 
tions and in all details. On Linux, Kdenlive [2] is a 
good choice for video editing: Mature, yet relatively 
simple in its basic approach, it is reasonably easy to 
use. (See the "Kdenlive" box for more information.) 


Preliminary Work 
A quick glance at Kdenlive's interface initially 
frightens off many potential users because of its 


The KDE project has been continuously de- 
veloping Its KDE Non-Linear Video Editor 
(Kdenlive) since 2002. With a history reach- 
ing back nearly 20 years, Kdenlive, like Gimp, 
is a Linux veteran. The few developers re- 
peatedly manage to put out new versions, 
but they don't always keep all the features 
from earlier releases. Plugins from other proj- 
ects are used for many effects and additional 
functions and sometimes have to be in- 
Stalled manually. The current version 21 
(more precisely, 21.08) is available in the re- 
positories of all the popular distributions. If it 
is missing in the repository of the distribution 
you are using, use the Applmage [3] instead. 
Alternative video editing programs for Linux 
such as Olive [4] or Shotcut [5] are currently 
the talk of the town but (still) offer signifi- 
cantly less performance-wise than Kdenlive. 
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Figure 1: A typical Kdenlive startup: Of the five program modes shown at the top right, Editing is initially the most impor- 


tant. 
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apparent complexity (Figure 1). But there is no 
reason for that at all: Of the five view modes (Log- 
ging, Editing, Audio, Effects, and Color) that Kden- 
live offers for selection in the upper right corner, 
only the Editing mode, in which you will create and 
edit your future video, is of interest for now. 

Before checking and changing settings, it is a 
good idea to make a copy of the configuration 
file found in your home directory below .config/ 
kdenliverc/. 

Next, you may want to adjust the theme used 
by Kdenlive so that it clearly color highlights the 
presets in the dialogs. To do this, in the Settings 
menu, for example, select KvDarkRed instead of 
Standard as the Color scheme and change the 
Style to Fusion instead of Standard. 

The optimal settings for your system depend on 
the desktop environment used as well as the 
themes provided by the distribution. Sometimes 
the selection of a certain color scheme or style 
also leads to completely unreadable menus and 
dialogs. In such a case, simply copy back the pre- 
viously saved configuration file or delete it alto- 
gether if necessary. 

Kdenlive has an extensive dialog with presets for 
the program (Figure 2). Settings | Configure Kdenlive... 
Supports extensive configuration of the program. 
Many of the normally quite useful settings should 
only be changed if you know exactly what impact 
they will have. 

There are also some defaults that dont make 
very much sense for slideshows. In Settings | Con- 
figure Kdenlive ... | Misc you will also find presets for 
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the display duration of images (Image clips), blank 
pages (Color clips), and faded-in titles (Title clips), 
among others. 

In Kdenlive jargon, clips mean all short video se- 
quences, regardless of how they were created. 
For slideshows, this also applies to the images 
you add. For images, a default of five seconds is 
clearly too long, and the program Is also too gen- 
erous with its three-second transitions. Fades re- 
fers to the fading transitions at the beginning and 
end of images or clips. 

Besides this, you will also want to adjust the 
window in Editing mode. It shows two preview 
areas by default, the left one (Clip Monitor) is nor- 
mally not used for slideshows and only unneces- 
Sarily reduces the space for the second preview 
area, the Project Monitor, where Kdenlive shows 
the project created so far. 

In the View menu, the individual components of 
the interface can be switched on and off. Here you 
also disable the (sometimes default) Time Remap- 
ping dialog and possibly the Project Notes (below 
the remaining preview area). However, it is not 
enough to just turn off the component, because 
Kdenlive will continue to keep the corresponding 
areas open. It will not release them until you dis- 
able the Library item in the View menu. Then you 
can place dialogs displayed elsewhere there. 

Once you have set up a layout, you can select to 
keep it for future sessions via View | Save Layout, 
where you can then set it up again with Load Lay- 
out. To do this, however, you first need to add it to 
the list of known layouts using Manage Layouts. 


Configure — Kdenlive 
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Open last project on startup 
Vv Activate crash recovery (auto save) 
Project Defaults v Check if first added clip matches project profile 
v Use KDE job tracking for render jobs 
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Image clips 00:00:05,00 
Image sequence 00:00:00,01 
Fades 00:00:03,00 


Subtitles 00:00:05,00 


© Cancel 


Figure 2: Before you create a slideshow, you need to adjust the preferences in Kdenlive. 
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First Steps 

To get started, first create a New Project using the 
New button in the toolbar. This creates the admin- 
istrative structures that Kdenlive needs internally 
and specifies the resolution and frame rate at 
which you want the application to create the re- 
sults. Kdenlive creates an extensive XML file for 
this purpose with a default extension of .kdenlive. 
The file forms the basis for all clips and images 
used in the following, which Kdenlive automati- 
cally converts if necessary. 

The first image displayed in the slideshow has a 
special significance. If it is too large, Kdenlive ad- 
justs the resolution used in the project by enlarg- 
ing it accordingly. This cannot be undone, at least 
not in a simple way. So be sure to use the resolu- 


Scaling and Moving Images 


If you can deliver the images used for the 
Slideshows directly with the correct aspect 
ratio and resolution, life is easy. Otherwise, 
you have to first fit and crop the images in the 
Slideshow. This at least offers the advantage 
that the “Ken Burns effect” and zooming can 
be applied. (The Ken Burns effect is a tech- 
nique — made famous by the US documen- 
tary filmmaker of the same name — of using 
slow pans and zooms and dissolves to create 
a video from still images.) In Kdenlive, your 
only option is to apply an effect. The context 
menu of the timeline offers the option to di- 
rectly access the most important effects with 
Insert an effect ... (Figure 4). 


For scaling, Zooming, and panning, use the 
Transformation effect, which you configure in a 
separate dialog box (Figure 5). Not all the set- 
tings there are immediately obvious. In the cen- 
ter of the window are four fields for entering 
displacements (X and Y) and for width and 
height (W and H). The row of buttons below 
quickly sets certain placements that are 
needed over and over again, adjusting the 
image to the height or width of the output, for 
example. Often the Adjust to width button saves 
further manual adjustments. In the Y field, you 
then set the desired position of the section in 
height. To do this, hold down Ctrl and move the 
displayed section with the mouse wheel. 


Movements in an image can be created just 
as easily: Move the section as desired (this 
works in the preview window with the 
mouse) and set a keyframe (see the “Key- 
frames” box). You can repeat this proce- 
dure as often as you like. Kdenlive will now 
automatically calculate the frames it needs 
to pan between. 
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tion you're aiming for in the finished video for the 
first frame. 

The default (and most useful in most cases) is 
Full HD resolution at 25 frames per second (Fig- 
ure 3). Higher frame rates result in some output 
devices being unable to display the videos. This 
setting can be changed later, but this involves 
time-consuming readjustment of the effects. 

After creating the project, load the images (and 
clips) that you want to appear in the finished video 
into the project window in the upper right corner 
and drag them from there to the timeline. To do 
this, the Add Clip or Folder button is available in the 
Project dialog. But there is another way. You can 
drag and drop both images and clips directly from 
the file manager into the timeline, and Kdenlive 
will then automatically display them in the Project 
window as well. Images appear, depending on the 
preferences, as short video sequences of the 
length predefined in the Settings menu. 

The Project window shares space with four other 
equally useful windows, which can be switched 
using the tabs below the window. Compositions con- 
tains a list of transitions you can use between clips. 
Effects shows effects that are applied to clips. Clip 
Properties lists the size of images and clips, as well 
as other metadata. The Undo History lists all the op- 
erations performed so far in the project. 

The length of a clip can be adjusted directly in 
the timeline. Use the mouse to move both the 
start and end points virtually at will and change 
the length of the clip. All images and clips that are 
not included in the slideshow can also be modi- 
fied using effects. The box “Scaling and Moving 
Images” shows how to do this using a frequently 
used example. 


Timeline 
The timeline is where the compositing and editing 
actions take place. This is where you add transitions 


Project Settings — Kdenlive 


Settings Proxy Metadata Project Files Cache Data 


Project folder to store proxy clips, thumbnails, previews 
Custom project folder 
Use parent folder of the project file as project folder 
Select the profile (preset) of the project 
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. _ Frame rate: 25 fps 
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> P§ SD/DVD Widescreen 
> BS} Custom 


“ Video Settings 


ite [-lemag-(e1 4) Audio tracks Audio channels 
Thumbnails: V Video Vv Audio 
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cancel | YOK | 


Figure 3: When creating a new project, you need to set the 
resolution and frame rate, among other things. 
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Figure 4: Effects can be inserted directly via the timeline’s context menu. To do this, activate Figure 5: The Transform dialog is complex; however, many 
the clip and then the context menu. For scaling, Transform is the right effect. switches have tooltips. 


and effects, though not yet defined in detail — this is done in special dialog boxes.  (RRAAAAMeaiaiatediaMaaaD alli 
The timeline is far more complex than it seems at first glance. — 
By default, Kdenlive creates a timeline with four tracks: two video and two 


audio. Each one stores only data of the corresponding type. A context menu este El linear > = 00:00:62,19 * 
lets you add more video and audio tracks to the left edge of the timeline at any x Sid teyfane 3840 GOH 2560 
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time, if needed. Images and clips can be moved around as you like with the 
mouse, and you can also switch between tracks. It does matter whether a clip 
is in the top or bottom track. This is evident in transitions, where normally the bande ITs 
clip in the upper track is on top of the one in the lower track, obscuring it. Never- [RE 
theless, it is not necessary to create a new track for each clip. ———— : 

The time markers displayed in the timeline refer to the finished movie, un- atc ins el, sc 
like those in the dialogs, which Kdenlive measures relative to the clip. You can Tete = & 
adjust the speed — and thus the display duration of the clip — at the edges. If Figure 6: The keyframe functions are available via special 
you apply an effect to a clip, its soeed decreases at the same rate as the clip's buttons below the mini timeline. 
length increases. The Change Speed function in the clips’ context menu 
works in the same way. It also supports pitch compensation for audio tracks. colaga incr peda cleats Sait 
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point for the effect. In Kdenlive, only some of the available effects sup- al Discrete Default keyframe type 
port keyframes, but they are quite easy to use. Compo 4 Linear Remove all keyframes after cursor 


Distort @ smooth 

The Transform dialog supports keyframes and displays the functions nana ETE a ; 

for them in a separate buttonbar (Figure 6). Use the X and Y position SgGRmrm = a 

fields to specify the desired section, Zoom in using Size, and use the Figure 7: The hamburger menu (here to the right of 
switches to set the image to the correct width, for example. Now place Smoothing) provides important functions for working with 
the mouse pointer at any position on the mini timeline in the dialog box keyframes. 

and use the Add Keyframe button to set a new keyframe with the cur- 

rent settings. The quickest way to set keyframes is to double-click on o—, (= Normalmode'y) 7% |b) % 00  00:00:64,12 / 00:00:67, 1 
an empty position in the mini timeline. Double-clicking on an existing —— ———— —— 


keyframe deletes it. ) ee ; | 
VA v2 ubuntu-default-gre\ 





You often want to copy keyframes to reapply the effects associated with 
them elsewhere. To do this, first select the keyframe you want to dupli- 
cate (Source). Then move the cursor to the position on the timeline 
where you want to paste the keyframe (target). Now click the Duplicate 
selected keyframe button (the fifth button). 


A number of important functions for keyframes can be found in the 
hamburger menu to the right of the keyframe type selection (Figure 7). 
The meaning of the different keyframe types is explained in the Kdenlive 
manual. 
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Figure 8: The fade can be precisely adjusted using the 
“handles” attached to the upper edge. 
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Figure 9: Cross-fading from one track to the other: The first transition is from track 2 (top) 
to track 1, the second vice versa (Wipe mode /nvert). 
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Rendering — Kdenlive 
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max-intra-rate=1000 
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Figure 10: For rendering, Kdenlive offers many options that only become visible with More 
Settings. 
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Imagination: 
http://imagination. 
sourceforge.net 
Kdenlive: http:// 
www.kdenlive.org 


Kdenlive Appimage: 


https:// 
binary-factory.kde. 
org/job/Kdenlive_ 
Nightly_Appimage_ 
Build/ 

Olive: https://www. 
olivevideoeditor.org 


Shotcut: https:// 
shotcut.org 


You have the option to arrange clips in the time- 
line, move their position, transport them from one 
track to another, or drag new clips directly from 
the file manager to the timeline. You delete them 
either with Del or via the context menu. This also 
applies to effects and transitions applied to clips, 
which you select up front for this purpose. You 
also need to manually select a specific effect or 
transition each time you want to adjust it. A col- 
ored frame indicates which element in the time- 
line is currently active. 

By default, the clip’s length can be shortened 
but cannot be increased beyond the original 
playing time. This can only be done by holding 
down Ctrl: Kdenlive now inserts additional cop- 
ies of the images that are evenly distributed 
through the clip to create the new length. This 


is not a problem with still images, because it is 
not visible in the result. When zooming, how- 
ever, it is noticeable as jerkiness in the zoom 
motion. 

One of the most useful non-obvious features is 
the fade-in and fade-out at the beginning or end of 
a clip. There are markers at the top corners of 
clips that you use to enable and adjust the fade ef- 
fect (Figure 8). 

Normally, to test the finished movie (or parts of 
it), you place the cursor at the top of the timeline 
and then press the spacebar to start playing. Al- 
ternatively, you can move the cursor directly in the 
time scale of the timeline and find the desired po- 
sition more quickly. If you have included clips, it 
takes a while for Kdenlive to actually display the 
current image. 

Transitions are a special type of effect, usually 
used at the beginning or end of a clip to cross-fade 
to another track that has another clip on it. The most 
important transition for slideshows, Wipe, can be ad- 
justed very precisely. To do this, apply the masks de- 
fined in the Wipe method to the clips to cross-fade 
the bottom track into the top track, or — with Invert 
active — vice versa (Figure 9). 

In the last step, Render produces the finished 
video (Figure 10). If you have chosen a manage- 
able output, this should not cause any problems. 
On most devices, the MP4/H.264/AAC combina- 
tion should produce good results. 


Conclusions 

Imagination, which | mentioned at the beginning, 
is fine for simple slideshows, but Kdenlive is in a 
different league. You can really control all the ef- 
fects precisely with Kdenlive and rework the finest 
details. It might make sense to combine the two 
programs. In this case, you first create smaller 
clips with Imagination, which you then connect in 
Kdenlive. For really complex effects and transi- 
tions, Kdenlive offers far more possibilities than 
other programs. It is definitely worth taking a 
closer look at the Wipe transition and the Transfor- 
mation effect. Sam 
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FILESHARING TO GO 





If you want to exchange files over the local network, you do not necessarily have to 
set up a file server such as Samba. A number of handy tools let you drag and drop 


to send files. 


Android 
android@Thomasf] gay 
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Figure 1: Warpinator automatically lists all active instances of the program on the local net- 


work in the application window. 
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ransferring files from one computer to an- 

other usually requires the installation of 

some kind of server. Samba (or a network 
share, in Windows speak) is the classic approach 
to doing this. You can also use SSH to move files 
from one computer to another quite easily. But if 
you dont want to install and set up a server, the 
only option is to use a USB stick since very few 
desktop environments integrate tools for easy 
data transfer. 

As an alternative to this, there are a number of 
Simply designed programs that transfer files and 
folders over the local network at the push of a 
button without requiring an Internet connection. 
And the system does not rely on background ser- 
vices or cloud servers such as commercial ser- 
vices like Dropbox or Google Drive for this. Once 
installed on the desired computers, the programs 
automatically find each other in the network. 
Sending data is then a matter of a few mouse 
clicks. | will look at the current crop of candi- 
dates for this task. 


i ETaitisels 
Send and Receive Files across the Network 
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Warpinator 

Warpinator [1] comes from Linux Mint’s rich reper- 
toire of standalone developments but can also be 
installed on other distributions. Linux Mint auto- 
matically integrates the program into its default 
installation. Arch Linux and Manjaro have the ap- 
plication in their regular package sources. On 
other systems, the easiest way to install Warpina- 
tor is as a Flatpak, but this involves numerous de- 
pendencies [2]. Optionally, the project’s GitHub 
page describes how to install from the source 
code [3]. 

Started from the application menu, Warpinator 
comes up with what is initially quite a clear-cut 
window. As soon as you launch the program on 
other computers in the local network, it lists these 
computers with the associated user and com- 
puter names (Figure 1). Click on one of the entries 
to open the detailed view. Use the asterisk next to 
the IP address in the header of the window to 
mark the corresponding computer as a favorite. 
This means that the entry will always appear in 
the upper area of the Warpinator instances acti- 
vated on the network. 

To send a file, click on the desired target and se- 
lect the desired file in the Send | Browse... dialog. 
Optionally, select an entire folder and then click the 
Add button below the file dialog. Warpinator will 
then transfer the directory along with all the files it 
contains. An even faster way is to simply drag the 
desired files into the File Transfers window. 

On the target side, the user sees a message 
about incoming files. Clicking Accept starts the 
transfer, and the Reject button cancels the ac- 
tion. In the default configuration, Warpinator 
saves the transferred data to the ~/Warpinator/ 
folder in the current user’s home directory (Fig- 
ure 2). The folder can be customized in the appli- 
cation's settings. This is also where you define 
how Warpinator will handle existing files and 
configure the ports on which the application 
communicates on the network. 


Teleport 

Teleport [4] minimizes what is already a very re- 
duced structure in Warpinator to the bare essen- 
tials - a compact application window, a simple list 
of currently active devices, and a Send File button 
to start the file transfer (Figure 3). There are no 
further settings, and don't look for file manager in- 
tegration or, say, the ability to transfer entire fold- 
ers. The minimal range of functions is not surpris- 
ing once you understand that the program origi- 
nates from the Gnome cosmos, where the devel- 
opers have been trying to reduce applications to 
the bare essentials for quite some time. 

If you like this extremely minimalist approach, 
the easiest way to install Teleport is to use a 
Flatpak [5]. Regular packages do not exist due to 
the still very young stage of the application's de- 
velopment. Not even the Arch User Repository 
(AUR), which is otherwise an almost inexhaustible 
source for installing the latest software, has a 
recipe for installation. Optionally, the developers 
describe on the project page how to compile the 
source code on Arch Linux and Ubuntu. Variants 
for other operating systems, as well as further 
functions such as encrypted data transfer or 
sending multiple files and folders, are on the de- 
velopers roadmap. 


LAN Share 
LAN Share [6], another open source app, follows a 
similar approach to that used by Warpinator. Ina 
direct comparison, the biggest visual difference is 
the toolkit. While Warpinator is based on GTK and 
thus integrates perfectly with the Gnome or Xfce 
desktop, LAN Share uses the Qt toolkit. The appli- 
cation thus makes the best possible use of the 
KDE Plasma desktop’s feature set. And this makes 
LAN Share easier to port to Windows. 

Although the first release of LAN Share was 
almost five years ago, no distributions have in- 
tegrated packages for LAN Share into their 
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Figure 3: Teleport reduces Warpinator's approach to the 
bare essentials. Currently the program only lets you 
transfer single files. 
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Figure 2: The history maintained by Warpinator for each client provides information about 
the status of each file and folder transfer. 


package sources thus far. On Arch Linux and 
derivatives such as Manjaro, there is the option 
of conveniently installing the program from the 
AUR. The developers offer deb packages [7] for 
users of Ubuntu and Debian. In addition, the 
project provides AppImages that work on all 
popular distributions [8]. 

For Windows, there is a setup with a standalone 
installation routine. In the test on a system with 
Windows 10, however, the program failed to 
launch because of a missing DLL file (MSVCR128. 
d11). However, installing the Visual C++ Redistrib- 
utable Packages for Visual Studio 2013 [9] solved 
this problem without too much overhead. 

To send afile, you need to install and start the 
program on all participating computers. Unlike 
Warpinator, LAN Share does not immediately list 
all the detected computers in the application win- 
dow (Figure 4). The potential receivers do not ap- 
pear until you click on one of the Send files... or 
Send folders... options and select an object to 
send (Figure 5). Once you select the target and 
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Figure 4: The LAN Share main window lists the previous transfers but not the currently 
active clients on the network. 
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press Send, LAN Share starts the transfer. If nec- 
essary, the transfer of larger volumes of data can 
be interrupted and resumed later on using the but- 
tons top right in the application window. 

By default, LAN Share writes the data to ~/LAN- 
ShareDownloads/ in the current users home direc- 
tory. You can adjust the folder using the settings 
found in Settings | General, if so desired. The 
choice of options here is limited to the bare essen- 
tials. At this point, you cannot configure anything 
apart from the behavior when files with the same 
name are found and which network ports to use. 


Conclusions 

Warpinator facilitates the fast transfer of data 
from one computer to another. However, com- 
pared to classical file sharing, the sender has to 
Start the transfer deliberately. The shared data 
is therefore not always available. And there is 
no Warpinator client for other operating sys- 
tems such as macOS or Windows as of yet. 
Having said this, there is already some discus- 
sion on the topic on the Linux Mint forum, and 
Android devices can now be integrated into the 
Warpinator network. The open source app for 


LAN Share 





Select Receiver — LAN Share 


Smartphones and tablets with Android is avail- 
able from the Google Play Store [10]. In our test 
involving a Google Pixel 5 with Android 11, the 
app worked without any complications. 

Teleport is only usable in the simplest cases 
thus far. The program still lacks many important 
functions, such as encrypted data transfer. Of 
course, the program's version counter had only 
reached 0.0.1 at the time of writing. 

LAN Share, on the other hand, performs just as 
well as Warpinator in practice. On a purely subjec- 
tive level, however, Warpinator make life a little 
easier, as it automatically displays the currently 
available targets in the application window. This 
means that you can see right after launching the 
tool whether the desired target is currently online. 
With LAN Share you would have to click Send first 
and then select a file or folder. On a positive note, 
LAN Share is available for Windows — which may 
be an important criterion for some users. BES 


[1] Warpinator: https:/github.com/linuxmint/ 
warpinator 

[2] Flatpak for Warpinator: https://flathub.org/ 
apps/details/org.x.Warpinator 


[3] Warpinator on GitHub: https:/github.com/ 
linuxmint/warpinator 





[4] Teleport: https://gitlab.gnome.org/jsparber/ 


teleport 
[5] Teleport Flatpak: https://flathub.org/apps/ 
details/com.frac_tion.teleport 
A irankiurt (ined [6] LAN Share: https://github.com/abdularis/ 
LAN-Share 


g [7] LAN Share releases: https://github.com/ 
abdularis/LAN-Share/releases 
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Figure 5: Users do not see the currently active targets for a data transfer until they initiate a 
data transfer. 
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Graham has finally been able to update the Raspberry Pi Zero in his 3D 
printer to a brand new Zero 2, unlocking webcam and print serving 
capabilities. sy GRAHAM MORRISON 
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Task manager 





Sparkling gems and new 
releases from the world of 
Free and Open Source Software 


Plasma System Monitor 





e look at lots of CPU, 
memory, storage, net- 
work, and process 


monitors in these pages. Popular 
as “first project’ for developers 
messing about with a new pro- 
gramming language or frame- 
work, theres a lot to gain from try- 
ing anew approach rather than re- 
lying on the old ways of using top 
or htop. However, we ve neglected 
other, more well-established sys- 
tem monitors that have improved. 
The best of these, KDE's Plasma 
System Monitor, was released 





more than a year ago to replace 
KDE's old system monitor and 
widgets and to take advantage of 
the new Kirigami UI! framework. 
The Kirigami UI framework's ef- 
fects and the design team’s excel- 
lent work are actually the first 
things you notice: Plasma System 
Monitor looks wonderful. Every- 
thing is drawn with vectors and is 
perfectly spaced and proportioned 
while also being responsive, re- 
gardless of how you size the win- 
dow or what proportion of the 
screen the application takes up. It 


1. Multiple pages: You can't change the first few pages, but you can add as many new 
pages as needed. 2. Task monitor: Merge and view sensors (meaning something to 
monitor) in different ways. 3. Edit mode: Group sensors together and align them verti- 
cally or horizontally. 4. Display styles: View sensor data in a histogram, a pie chart, as 
text, as lines, etc. 5. Aggregate data: Join and merge various sensors. 6. Widget: A 
panel and desktop widget offers all the same features. 7. Data overload: Process and 
application lists include CPU, memory, network speeds, and storage usage. 
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will even work well on a smartphone. When first launched, 
the default overview page includes three rotary charts for 
memory, storage, and CPU usage; scaling the window size 
adjusts these automatically in real time with their contents 
dynamically updating as their sizes change. Each graphical 
element crams as much detail into the available space as it 
can without being overwhelming. The Applications list is 
thoroughly informative: For example, it shows every appli- 
cation running, along with its CPU and RAM overhead, and 
shows both its incoming and outgoing network use and its 
read and write storage throughput. 

Several other pages in the default configuration show 
more typical task manager information, including a page 
that lists all the running processes, with the same details as 
in the application view, alongside views to show parent and 
child processes, those owned by certain users, and the abil- 
ity to send any signal to any listed process. Theres also a 
beautifully rendered histograms page for CPU, memory, 
and network use. Beyond the default settings, Plasma Sys- 
tem Monitor is fully editable, allowing you to add and re- 
move pages, create horizontal and vertical containers, and 
add any number of monitoring sensors to create your own 
dashboard. Theres a huge list of sensors ranging from indi- 
vidual CPU and GPU cores to storage, operating system 
Statistics, memory, and network details. All of this can be 
rendered using a variety of display styles, including pie 
charts and histograms, along with tables, grids, lines, and 
even simple text, and encapsulated within a desktop widget 
you can add to the desktop or panel. 

This is reminiscent of that other great KDE monitoring 
tool, KSysGuard, which is still being maintained. KSys- 
Guard is similarly modular and can even monitor remote 
servers, but it's also harder to use and starting to look its 
age. Plasma System Monitor is agile by comparison. It can 
often feel more like an IDE for a modern web and phone 
framework (Such as Home Assistant's dashboard editor) 
than a task manager, which Is precisely why Plasma Sys- 
tem Monitor is so good. 


Project Website 
https://invent.kde.org/plasma/plasma-systemmonitor 





PDF reader 


Sioyek 


here are many PDF view- 
ers for Linux and several 
of them are good if not 


exceptional. Okular, in particular, 
is one of the best we've used, be- 
cause it's one of the few that will 
open most PDFs, offers sensible 
scaling, multiple page views, and 
the ability to annotate PDFs ina 
way that can be shared with other 
people. If you need to manage a 
library of PDFs, then the ebook 
editor and manager, Calibre, is an- 
other excellent and more ambi- 
tious choice. It's particularly good 
when youre researching a subject 
and need to search across several 
publications, although Calibre's 
PDF-viewing capabilities are little 
more than perfunctory — which is 
where Sioyek can help. Sioyek is a 
new PDF reader developed spe- 
cifically to help better manage 


Audio visualizer 


projectM 


here are few time-wast- 
ing utilities as satisfying 
as amusic and audio vi- 


sualizer. And while they can turn 
what should be a purely listening 
experience into a gluttonous 
audio/visual feast of migraine-in- 
ducing proportions, they can 
equally enhance the listening ex- 
perience, especially at a party or 
at a live performance. But even 
on the humble desktop, they're a 
lot of fun, and projectM is the 
best visualizer we've come 
across in years. A music visual- 
izer is really any kind of visual ef- 
fect that attempts to graphically 
sympathize with the sound that 
is pushed through it. They usu- 
ally involve lots of bouncing lines 
syncopated to a beat; or psyche- 
delic, pulsing wheels of color 
turning against one another, or 


and access academic papers, but 
it's equally adept at beautifully 
rendering PDF content, especially 
if the files contain formulas, and 
for searching across a collection. 
Almost like a Vim PDF viewer, 
Sioyek is controlled entirely from 
the keyboard with a small com- 
mand console that appears 
when you type something. You 
can quickly search across any 
file you've previously opened, as 
well as across the table of con- 
tents and even referenced fig- 
ures and bibliography entries 
when they're not in the table of 
contents. There are commands 
for inserting a bookmark, so you 
can quickly get back to the same 
section after chasing another ref- 
erence, and a more formal book- 
mark manager which can be 
used globally to reference sec- 


everything combined. They can 
be an old-school Amiga demo, 
part tech demo, and also an ex- 
periment in just how many mil- 
lions of particles your GPU can 
push onto the screen. Their suc- 
cess Is entirely subjective and 
often linked to the listening envi- 
ronment and motivations for 
using one. Which is why they're 
often configurable. 

projectM does all of this and 
much more. It can run as a stand- 
alone application with selectable 
input, as a PulseAudio effect, as a 
plugin from the Kodi media cen- 
ter, and as a library to use from 
your own projects. The whole 
project itself is a reinterpretation 
of MilkDrop, a relic visualizer for 
the old Winamp Windows music 
player, and it uses a library of pre- 
Set files to create the visual ef- 
fects. These files tell the pixel 
shaders what to do using various 
equations and parameters, and 
projectM includes those from 
MilkDrop as well as an enormous 
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Sioyek is a PDF reader designed primarily for accessing a library of 
academic papers, but it's also excellent as a general reader. 


tions in any document. You can highlight a section of text, 
which is then separately searchable, and even open an- 
other reference while the original document is still open. 
It's an excellent application, with both light and dark 
modes. While it's ideal for academic searches, it’s also 
perfect as your default PDF reader. 


Project Website 
https://github.com/ahrm/sioyek 





Recreate 1990s rave culture by projecting the output from projectM 
onto a nearby wall. 


library of its own containing 41,000 other presets. There 
are controls to switch between these and also a random 
preset, as well as the ability to create a playlist of presets 
and lock to the currently selected preset in place. You can 
also increase and decrease the beat sensitivity so that you 
can ensure the pulsating blob of pixels is always in time to 
your music. It's a brilliant distraction and perfect for parties 
(but not great with a hangover). 


Project Website 
https://github.com/projectM-visualizer 
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Downloader GUI 


yt-dig 


f you've tried browsing the In- 
ternet recently without some 


form of ad blocking, it can be 
a real shock. There are pop-ups 
everywhere, invasive Al chats, en- 
dorsements inserted into text, site 
takeovers, slide-downs, slide-ins, 
and every other kind of invasive, 
attention-grabbing mechanic you 
can imagine. It's truly a sorry state 
because many great content sites 
— such as our own — obviously 
rely on advertising to fund the 
work they do, but the Wild West of 
click-throughs has broken any 
possible kind of compromise. Be- 
cause most people don't run any 
kind of ad blocker, they must think 
the Internet is a very different kind 
of place than those of us who do, 
which perhaps explains why they 
may also be unconcerned about 
other aspects of the Internet, 


Profiler and monitor 


Guider 


f KDE Plasma’s System Mon- 
| itor is the most beautiful 
monitoring tool we've looked 
at, then Guider is a good candi- 
date for the most comprehen- 
Sive. It doesn't even consider it- 
self a monitoring tool, preferring 
the term “integrated perfor- 
mance analyzer, which is a fair 
description. Its list of features in- 
cludes monitoring, profiling, trac- 
ing, visualization, control, log- 
ging, networking, and testing, all 
of which can be conjured from 
the command line. Guider is writ- 
ten in Python and is easily in- 
Stalled with pip. At its simplest, 
you can run guider top -ato 
launch the tool as a simple top 
process viewer, although simple 
isnt really the right word. Every 
monitoring mode contains a 
huge amount of data, and it can 
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such as a lack of privacy or ag- 
gressive personal profile building 
to trick you into spending more 
time viewing and scrolling. 
YouTube is becoming one of 
the worst sites for this, with both 
invasive advertising you can no 
longer skip and addictive further 
recommendations that can turn 
30 seconds of research into 30 
minutes on cats making beats. A 
decent blocker can help, and 
there are Firefox plugins that will 
also hide the recommendations, 
but another good alternative Is to 
only watch the videos you want 
outside of YouTube's tempta- 
tions. This is how the wonderful 
youtube-dl script has become so 
popular: by letting you access the 
often incredible content you find 
on YouTube without so much of 
YouTube. But it's a command 


easily overwhelm your terminal. 
The top implementation, for ex- 
ample, is wide, listing statistics 
for every core alongside a pro- 
cess table and general system 
overview. 

Other modes can be used to 
analyze processes, measure the 
running time and resource usage 
of specific functions, introspect 
the call stack, and perform signal 
tracing. You can even reconfig- 
ure the scheduler for a process 
to give it more or less CPU time, 
or limit its CPU usage, all from 
the same command, although 
this does sometimes depend on 
your kernel configuration. There's 
often a lot of output, which can 
optionally be piped into a sepa- 
rate JSON-formatted file or even 
viewed from a web browser. 
Threads can also be traced, re- 
corded, and their history played 
back and even turned into histo- 
grams and charts directly from 
within Guider itself, although 
you'll want to view them from 
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Stack up URLs, see their ETAs, and limit your own viewing bandwidth 
with this excellent YouTube download GUI. 


line-only solution, and that leads us to this dis- 
covery, yt-dlg, a fully maintained and updated 
GUI for the command-line downloader that 
turns YouTube viewing into a simple drag-and- 
drop experience. The URL for your target is 
simply added to a list, where you can then see 
how much bandwidth it's taking and when it 
will be downloaded, allowing you to view it di- 
rectly or from your favorite application. 


Project Website 
https://github.com/oleksis/youtube-dl-gui 





Guider is a monitoring tool that goes from simple process monitor- 
ing to thread tracing and profiling. 


your favorite browser or image viewer. 
There's a lot to learn, but the tool is also 
Straightforward and easy to use for simple 
monitoring, making it ideal for someone who 
needs something that will grow with them as 
their knowledge increases. 


Project Website 
https://lipeace.github.io/docs/guider.html 


Share URLs 


utm_no 


hen Tim Berners-Lee 
invented the URL back 
in 1994 he could 


never have anticipated that the 
humble uniform resource locator 
would become so easily sub- 
verted. Back then, a URL was 
nothing more than a static web 
address for an actual HTML 
page that either existed or didn't. 
There were assumptions, such 
as the root of a location being 
served by an index.html file, but 
most pages were static, written 
by hand, and didn't change loca- 
tion. They certainly werent dy- 
namic or served via a content de- 
livery network. All of this has 
changed in the decades since. 
Most people laugh at hand-writ- 
ten HTML now, and even if you 
wrote it, you probably couldnt 
persuade your browser to load it 


Dune 


ver the years, weve 

looked at many different 

shell environments. The 
vast majority of these attempt 
to make you more productive ei- 
ther by mimicking the way you 
might work with a programming 
language, such as with Micro- 
soft’s PowerShell, or old Unix 
systems with Bash, or even 
modern search-based shells 
such as fish. None that we've 
looked at so far have attempted 
to make you feel more “cozy,’ 
but that’s exactly what Dune 
has been developed to do. Cozy 
in this sense does need a little 
qualification, because it’s not 
the kind of cozy you might get 
from lounging around an open 
fire with a good book. Instead, 
it's the kind of cozy you might 
feel if you were a developer who 
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without carefully wrapping it 
within HTTPS. Most web pages 
are now generated programmati- 
cally from content management 
systems using a variety of pro- 
gramming languages and frame- 
works, and their URLs are gener- 
ated the same way, even when 
the endpoint where users access 
them remains the same. 

The same is also true of how 
those URLs are shared and discov- 
ered. Search recommendations, 
embedded email links, social 
media posts, and message chat- 
ting services are all now guilty of 
augmenting URLs with their own 
added metadata — to both better 
track you and to inform the onward 
server who to be thankful to for the 
new reader. These bits of added 
metadata are known as UTM 
tracking links because they're usu- 
ally separated from the remainder 
of the URL by utm_, followed by an 
arbitrary parameter name. UTM 
links obviously have huge implica- 
tions for privacy, which is why 


wants your command line envi- 
ronment to behave more like 
your IDE. 

Dune is described by its author 
as a ‘shell by the beach,’ devel- 
oped while the author was bored 
at college. The first thing you see 
is aneat ASCII art banner with the 
time and date, followed by a brief 
introduction into the shell itself. 
This intro will step you through 
some of its unique and not-so- 
unique features, including mac- 
ros, variables assigned with a let 
commana, and the built-in help 
system. The features that should 
help a programmer feel cozy in- 
clude functions that can be over- 
loaded to create your own cus- 
tomized experience, including the 
various prompts and output re- 
ports. Theres also a standard li- 
brary of functions that are ac- 
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kryogenix.org/code/utm no 


Stuart Langridge 


Licence 


Not only isutm_no a useful tool in itself, the code is easy to under- 
stand, especially if you want to see how to implement tests in your 
own code. 


many of us try to always manually edit them out. But with 
this brilliant little tool, you dont even need to do that. utm_no is 
a small utility that sits in your panel and transparently waits 
for a URL to appear on the clipboard. When it does, it equally 
transparently strips that URL of the UTM tracking metadata 
so that when you come to paste It, only the URL elements re- 
quired are included. It's simple but brilliant. 


Project Website 
https://www.kryogenix.org/code/utm_no/ 





Dune isn't even the first shell developed by its author, who also cre- 
ated Atom, which Dune attempts to improve upon. 


cessed by typing math, showing both useful values and the 
functions you can use in your own scripts. If you don't use 
any of these features, Dune will operate just like Bash, only 
with a slightly different prompt and color scheme and 
more fun elements such as widgets. But if you do start to 
modify elements of your environment, Dune makes it a lot 
of fun and, yes, it can even feel quite cozy. 


Project Website 
https://github.com/adam-mcdaniel/dune 
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Animation tool 


VPaint 


he best drawing tools 
seem to have the word 
“paint” in their title. There's 


the famous Deluxe Paint that 
helped sell a million Commodore 
Amigas in the 1980s, and regular 
PCs in the 1990s, to such an ex- 
tent that its pixelated Tutankha- 
men and Venus images are now 
synonymous with the dawn of 
high-quality graphics composition 
on home hardware. Deluxe Paint 
also helped many of us adapt to 
what might have been our first 
touch of a mouse, left-clicking to 
draw and drag those pixels across 
our flickering CRT televisions. 
Later came Photon Paint with 3D 
modeling and texture mapping 
and then even CinePaint, which 
was an early fork of Gimp with real 
movie-making credits to its name. 
And now we have VPaint. 

The V in VPaint is for vector, 
which means that, rather than 
drawing in the pixels of Deluxe 
Paint, you draw in vectors, the dots 
and lines between different canvas 
locations. It's a very different kind 
of drawing application than the 
other well-known vector drawing 
tool, Inkscape, because VPaint 


feels instantly creative. When you 
first launch the application, you 
can right-click your mouse and 
drag the pixels across the screen 
just as you could with Deluxe 
Paint, but VPaint is doing lots of 
things in the background. Depend- 
ing on your skill, the end result will 
be either a natural-looking thick 
line, from one point to another, or a 
meandering, imperfect scrawl 


from which your therapist will likely 


draw conclusions about your 
childhood. Either way, it wont look 
like a typical straight line or pixel- 
lated wobble. Peak behind the can- 
vas, and all the lines, shapes, and 
curves you draw are vectors. You 
can see this most evidently with 
the Sculpt tool which, when se- 
lected, can be used to modify your 
doodles by dragging the cursor 
across the lines youve made, 
stretching them into new loca- 
tions. The cursor will stick to a cer- 
tain line that can then be pulled 
away from its original position. If 
theres more than one line at the 
point you've chosen and other 
lines connected within a circle of 
influence, all these will be dragged 
too, with the amount of deformity 





Drawing with VPaint is intuitive, and your lines are automatically smoothed to look 
natural. 
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Use the onion skin option to see other animation frames while you 
tweak the current one. 
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being relative to the distance from the 
original point. It’s like pulling at the 
threads in a woven fabric. 

For more advanced editing, there 
are layers to keep different parts of 
your illustrations separate from other 
parts, and it's also easy to group ele- 
ments together and raise and lower 
them within the object hierarchy of the 
image. This is all great for drawing 
and design and common to other 
drawing tools, but drawing isnt the 
main reason for using VPaint. That's 
because it also does animation. This 
is only noticeable if you wonder what 
the small chain of cells represent in 
the bottom of the main window. 
These are to hold the various states, 
or keyframes, you want your anima- 
tion to progress though. Vector im- 
ages are ideal for this because a for- 
mula can easily be used to calculate 
the exact tweening values between 
frames, making transitions between 
one keyframe and the next always as 
smooth as possible. This is how all 
modern animation is generated and 
even has a lot in common with an- 
other ancient Amiga original, an appli- 
cation called Fantavision. VPaint is 
Still in the relatively early stages of de- 
velopment and is currently classed as 
a prototype ahead of a potential com- 
mercial release, but it's being released 
under the Apache license, which 
means development should continue 
regardless. 


Project Website 
https://github.com/dalboris/vpaint 


Strategy game 


UnCiv 


id Meier's Civilization 

games are some of the 

best games ever made. 
They combine empire building 
and resource management with 
finely balanced strategy — and 
even some historical and eco- 
nomic contexts that make 
sense today. It's no surprise that 
remakes and other games in- 
spired by them often grace 
these pages. The original games 
Span the entire modern gaming 
epoch, from the pixelated Amiga 
graphics of the early 1990s to 
the retina displays of iPads, An- 
droid phones, and consoles in 
the current era, which means 
there's an aesthetic to Suit every 
remake. UnCiv is one of the 
best, taking inspiration from the 
current, penultimate Civilization 
release, Civilization V, originally 


Space survival 


released in 2010 but still very 
much alive. 

However, the inspiration that 
UnCiv has taken from Civ V isn't in 
the graphics but in the gameplay. 
Rather than the beautiful isomet- 
ric 3D graphics of the original, 
UnCiv goes back to the pure tiles 
of the original, albeit with lovingly 
created cartoon-style artwork. 
This doesnt look as good as a 
modern game, but it does mean 
UnCiv will run on almost anything, 
from an Android phone to your 
Linux desktop, and even the Rasp- 
berry Pi, thanks to some recent 
code patches. What you get is a 
brilliantly playable strategy game 
set within a hexagonal map with 
the same historical theaters, war- 
riors, settlements, specialities, 
and natural resources, and all the 
Statistics, you could ever want. All 
of this is being lovingly recreated 
rather than reverse engineered, 
which means it all feels the same 
but similar to the original, which is 
a good thing because the IP of 


Space Station 14 


pace Station 14 is a unique 

top-down multiplayer 

game set on a space sta- 
tion. It's a remake of a cult classic 
called Space Station 13, but this 
version is completely open source 
and still under active development. 
The developers consider the game 
to be in an alpha state, and it can 
feel that way when it comes to try- 
ing to understand what's happen- 
ing and what your overall objective 
might be. But the developers are 
also asking for people to test the 
game and contribute to the com- 
munity, so it's definitely worth giv- 
ing ita go. Once installed, you first 
need to connect to a server with 
other players, which will then 
download whatever assets are 
being used by that particular game 
instance. You then get to choose a 
character to play. There are many 


roles to choose from, including 
bartender, chef, clown, head of se- 
curity, and captain. Each role 
comes with its own gear and task 
list. You'll soon find yourself 
aboard a pixelated spaceship con- 
sisting of ahuge number of areas 
and, critically, lots of other players. 
To start with, the game can feel 
a little like Among Us. You appear 
on the space station with no 
knowledge of its layout and loca- 
tions or even your character's Ca- 
pabilities. You then point and click 
to move between areas and inter- 
act with other people through 
moveable chat windows. There 
are rogue-like elements to the 
graphics, especially in the fog-of- 
war effect that obscures the parts 
your character wouldnt be able to 
see from their current position, but 
the graphics are very effective at 
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Move a unit! 
Click on a destination > Click the arrow popup IT 
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If real civilization is proving a little disappointing, why not create 
your own with UnCiv? 


the original games are still likely being protected. This 
means UnCiv is a game for Civilization fans that can be 
played guilt free on your favorite operating system, regard- 
less of whether you've played the originals or not, which we 
highly recommend doing. 


Project Website 
https://github.com/yairm210/Unciv 





Around the border of the game window is a “hotbar’” for actions, short- 
cuts for controls, the chat window, and the items you're holding. 


creating an immersive space station-like atmosphere. It 
feels reminiscent of an old Amiga and Atari ST game called 
Pandora where you appear on a similar space station and 
need to solve a murder. In Space Station 14, the idea is that 
various disasters will befall the station and you and your fel- 
low players must work together to keep everything going. 


Project Website 
https://spacestation1 4.10 
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Prettymaps combines multiple Python libraries to make It easy to draw maps 
Straight from the OpenStreetMap database. 


FEBRUARY 2022 


BY MARCO FIORETTI aps are terribly important. Without 
maps we would not know where to go 


or, in many cases, what to do, either. 
Accurate maps are essential to planning anything 
— from a romantic weekend to large-scale urban 
projects. But maps are also beautiful in and of 
themselves. If they are customized by their owner, 
they can become little pieces of art. In this tutorial, 
| introduce a relatively simple way to generate 
such art using OpenStreetMap [1] and free and 
open source software. The results include maps 
in several styles and shapes that can be used as 
wall posters; illustrations in brochures and other 
documents; decorations for mugs, pillows, and 
other household items; or ... just be used on the 
road, aS good old paper maps! 


OpenStreetMap 

If you have used the Internet at all in the past 15 
years, very likely you already know what Open- 
StreetMap is. But just in case you dont, here is a 
Super-quick definition, focused on the aspect that 
is directly relevant for this tutorial: OpenStreet- 
Map (OSM) is the Wikipedia concept applied to 
digital cartography, an online map of the whole 
world, built and continuously updated by thou- 
sands of volunteers from all around the world. It is 
hard to explain how important OSM is for our so- 
ciety, but anyone interested may find food for 
thought on this matter in my blog [2]. 

Here, what really matters is the fact that all the 
OSM raw data is available for reuse under an open 
source license and can be downloaded automati- 
cally through equally open programming inter- 
faces. It's this availability that makes drawing cus- 
tom digital maps possible, even for non-program- 
mers and for commercial purposes. 


prettymaps 

The maps that you see in this tutorial have been 
generated with prettymaps [3], a Python package 
that can fetch selected raw data straight from the 
OSM database and then pass it to other libraries 
that will do the actual drawing according to your 
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instructions. In my experience, prettymaps is a lit- 
tle gem that can make map enthusiasts happy, in 
spite of a couple of drawbacks. 

The minor drawback is that prettymaps is slow. 
Drawing a relatively small map may take several 
minutes, even on an Ubuntu desktop with 16GB of 
RAM. Then again, this is software that many users 
will only run occasionally (and always in the back- 
ground, maybe while taking a nap), so it’s not re- 
ally a big deal. 

A bigger obstacle, at least as of this writing, is 
the fact that the only documentation is the raw 
code examples present on the Jupyter Note- 
book [4] and GitHub pages [5] of the author and 
other users of prettymaps. Don't despair, 
though! This tutorial explains how to install 
prettymaps and then how to modify any of 
those examples to draw any place you want, 
even if you have never used Python before. In 
order to get there, however, it is necessary to 
Start with a quick overview of the prettymaps 
workflow and main components, as well as 
some basic concepts of digital mapping. 


What Are Digital Maps? 
Under the hood, OSM and any other digital map 
are standard relational databases, with some 
extra data types and functions made-to-order to 
Support mapping. Such databases are usually 
called geographic information systems (GIS). 
How the data is actually presented to users (i.é., 
how any map built with GIS actually looks like on 
a screen) is delegated to separate programs. 
Maybe the first essential feature of any GIS is that 
all data is organized in layers, one (or more) for 
every terrain-related feature: streets, buildings, 
forests, water (rivers and lakes), railroads, shops, 
sewers, administrative boundaries, and hiking 
paths. All these features, and many more, have 
their own separate layer that users can show, 
hide, mix, or visually format as they wish. 
Second, GIS databases support storage and 
analysis of lines and polygons. This is what 
makes it possible to, for example, draw state or 
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city borders, calculate the length of an itinerary or 
the area of a building, represent the exact shape 
of a lagoon, or figure out which parts of a moun- 
tain range belong to one country or another. 

Third, each point of each element in the data- 
base may also have an elevation, its height above 
or below sea level. Among other things, this per- 
mits the user to calculate steepness of roads and 
draw 3D maps. 


The prettymaps Architecture 

At least from the outside, prettymaps seems lit- 
tle more than a small blob of well-done glue 
logic that connects many other Python libraries 
just for the purpose of drawing maps based on 
OSM data. The most important of these librar- 
ies are OSMnx, Matplotlib, Shapely, and vs- 
ketch. Any of these tools deserves a tutorial of 
its own, but here we only need to know the main 
features and role of each. 

OSMnx [6] can download any combination of 
layers and other data from the OSM database, al- 
lowing the user to select them by place name, 
placement inside a polygon, distance from a given 
address, or by geographical coordinates. Once it 
has fetched the data, OSMnx can also save It to 
local files in several formats or use it to calculate 
Street orientation, inclination, and many other 
characteristics. After OSMnx, prettymaps loads 
functions from Shapely [7] and vsketch [8] to pre- 
pare the raw drawing by analyzing and modifying, 
as requested, any geometric object in the data. Fi- 
nally, Matplotlib [9] creates and saves the actual 
image in several formats, with the zoom levels, 
captions, and visual style specified by the user. 

Last but not least, to work without problems, 
prettymaps also needs |Python [10], the Python 
command shell for interactive computing in Py- 
thon and other languages. 


Installing All the Pieces 

Configuring a Linux system to use prettymaps is 
not really difficult. However, it took me about 15 
minutes, much longer than anything else | have 
installed on my Ubuntu box in the past year, and | 
have to say that the installation is pretty hard to 
document as one linear procedure. The reason is 
simply that prettymaps needs lots of specific ver- 
sions of many other libraries, which may or may 
not be already installed on your system. Above all, 
depending on your distribution, those packages 
may not be available as native packages but only 
through unpredictable combinations of Flatpak, 
Snap, pip, or any of the other systems that have 
made Linux life much more complicated than it 
was 10 years ago. The good news is that you al- 
most surely wont have to compile anything from 
source, and that the whole process should not 
take more than 15 minutes. For example, on my 
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Ubuntu 21.04 computer, | had to type com- 
mands such as: 


sudo apt-get install -y ipython3 

pip install git+https://github.com/2 
abey79/vsketch#egg=vsketch 

sudo pip install prettymaps 


to install |Python, vsketch, and prettymaps itself, 

plus similar commands for the other libraries. In 

practice, the safest and most portable advice and 
procedure to get prettymaps up and running ona 
generic Linux box seems to be this: 

Install prettymaps as above, plus any depen- 
dencies it may demand right away. 

Copy and paste any of the examples from the 
website into a plain text file called, for example, 
prettymap-test. py. 

Open a terminal and type python3 pretty- 
map-test. py. 

© If there are any errors, figure out from them 
which library is still missing. Type How to install 
<library> on <Linux distribution> in a search en- 
gine and follow the instructions. 

Repeat the previous step until prettymaps ac- 
tually generates a map. 


A Basic prettymaps Script 

As already mentioned, once everything is installed, 
all you have to do to generate a map with pretty- 
maps Is type python3, followed by the prettymaps 
script file at acommand prompt. Figure 1 shows the 
circular map of Rome that | got when | did that with 


Gata @ Qpenitreettap contribaters 
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Figure 1: A map of Rome in 


circular format is easily 


derived from the prettymaps 


examples. 
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the code in Listing 1, which comes from one of the 
examples provided by the developer. | only changed 
the initial coordinates — and omitted all comments 
because | am going to explain the code here. 

Listing 1 is quite verbose but much less compili- 
cated than it may seem at first sight. The first 20 
lines just load all the libraries that prettymaps 
needs. Line 21 defines the Matplotlib figure that 
must be created, setting its axes and size. 

There are several more options that you may op- 
tionally set in this initial part of the script, starting 
with a title for the whole figure and a rotation and 


Listing 1: Circular Map of Rome 


offset, if, for whatever reason, you would like the 
map rotated at any angle or not at the center of the 
image. Also important is the “backup” option that 
allows you to load the results of a previous run of 
prettymaps to save time. For the corresponding 
details, please see the prettymaps website [3]. 
Possibly the core instruction of the whole 
script, line 24 describes what area of the whole 
Earth should be represented in the map. In this 
case, It is a circle with a 10km radius, centered 
on the latitude and longitude that approxi- 
mately correspond to the center of Rome. 


Ol 39 'pedestrian': 2, 
02 from IPython import get_ipython 40 'footway': 1, 
03 ipython = get_ipython() 41} 
O4 
42 }, 
O5 af '__ IPYTHON * in ellobals(); 
43 "building: {"tees': {'’bualding": Trae, “landuse’: 
O06 ipython.magic('load_ext autoreload' ) 
“Construction” +; "“undon"> False. 
O07 ipython.magic('autoreload 2') 
a 44 ‘water': {'tags': {'natural': ['water', 'bay']}}, 
ho) dmeers? Gree Gre oxen aomen 45 'green': {'tags': {'landuse': 'grass', 'natural': 
ae ['island', 'wood'], 'leisure': 'park'}}, 
11 from prettymaps import * 46 'forest': {'tags': {'landuse': 'forest'}}, 
12 import vsketch 47 'parking': {'tags': {'amenity': 'parking', 'highway': 
13 import osmnx as ox 'pedestrian', 'man_made': 'pier'}} 
14 import matplotlib.font_manager as fm 48 }, 
15 from matplotlib. 1mpert pyplot as plt 49 drawing_kwargs = { 
16 from descartes import PolygonPatch 50>" backeround' 94 £6". “+P2b4CR". "ce.  sdadsel".) ‘nate’: 
17 from shapely.geometry import * '900...', 'zorder': -1}, 
. . o * 
LS atu elee (ote Ey sieusae dae Lalhglonst Bl *perimeter': {'te": *#RAR4YCB”. “ec*? *#dadbel’, "iw": 0, 
19 from shapely.ops import unary_union 'hatch': 'ooo...', 'zorder': 0}, 
20 
be. (ereen! 2 he's {7 DOPIBE . “eC. “Fers737 . * lan a. 
alee ak ax = plt.subplots(fiesize = (12, 12), constrained , ee 
E> BP BP ale ( ) = zorder <li), 
layout = True) 
ae Bo “norest ¢ ('2e's “S64 B0GR", ‘eet. 'sera7a7 5  lwes 1. 
'ZOrgetre =) Alls 
23 layers = plot( 
SY “water: {"£e': “#alesft', “ee*: “F2R2737', ‘hHateh": 
24 '41.894988965651585, 12.491488590684028', radius = 10000, 
"OOO, 4, 3) MabCnee + e5cve>o 4 wie i 
oo ax, = ax, 
'ZOrdervs 2) 
26 layers = { 
: 5S ‘parking: 77 te “Fee PeCR 2) “eel i. “eR ava7 2) woes I 
27 'perimeter': {}, 
'zworder’: at. 
28 'streets': { 
. ; . 56 “Streets * (° tes "S2R3737'5 “ec. “#4875657"., ‘alphate 1. 
29 'custom_filter': '["highway"~"motorway|trunk| primary | t e @ e 
secondary |tertiary|residential | service | ‘Iw': 0, 'zorder': 3}, 
unclassified|pedestrian|footway"]', 57 'building': {'palette': ['#FFC857', '#E9724C', '#C5283D'], 
20 “widths 4 ‘ee's” “Sersyor . '“lwie «5S. “zorder’ + 4). 
si) “moterway': &, se)? 
32 URUK =, 65. 
ah) 
32“ pramary = 4.5. 
60 osmimeredit- =) color =) “Fersvsr 4 
34 ‘secondary’: 4, 
. 61 ) 
25) suerte SB ca. 
36 'residential': 3, Ge 
37 'service': 2, 63 plt.savefig('rome-circular.png' ) 


So “unelassitied”; 2, 64 plt.savefig('rome-circular.svg' ) 
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» OpenStreetMap 





Share 


Geographic coordinates to 
use in Prettymaps scripts 


Link or HTML 


Include marker 
Link 


https: //www.openstreetmap. org/#map=18¥41, 


Geo URI 


geo:41.89423, 12.48205?7z=18 


Image 


Set custom dimensions 


Format: PNG v 


Scale: 1: 1610 


how standard layer at 1007 x 909 


Figure 2: OpenStreetMap’'s Share dialog also lists the coor- 
dinates of the city center used in Listing 1. 


Finding the coordinates of whatever point on 
Earth you would like to map is easy. In OpenStreet- 
Map, double-click on the point of interest and select 
the Share icon from the right toolbar to open a form. 
Figure 2 shows the form with the coordinates to 
copy into the prettymaps file. On Google Maps, se- 
lect the point, right-click on it to open a pop-up win- 
dow, and copy the coordinates from there (Figure 3). 

Important note: The layers one may download 
with the rest of Listing 1 are so many, and with 
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SO many options, that describing them all might 
fill a book. Therefore, | only highlight the most 
important features and common characteris- 
tics of the corresponding function and suggest 
you ask for more details either on the pretty- 
maps GitHub page [3] or the corresponding 
subreddit [11]. 

After setting the position, shape, and extension 
of the map, lines 26 to 48 describe which layers 
should be downloaded from the database and 
how to select their sub-components if needed. Ex- 
amples of this capability are the custom_ filter of 
line 29 and the options for buildings, parking, and 
green areas later on. When applicable, for exam- 
ple, with streets, you may also specify the line 
width in line 30 that corresponds to any different 
type of “street.” 

Lines 49 to 58 specify how to draw and paint 
each layer, as it is evident when confronting all 
the sub-parameters of the drawing_kwargs func- 
tion with the layers listed in the previous block of 
code. Here, each layer previously called is given 
a foreground (fc) and edge (ec) color, as well as 
line width (lw), hatching (i.e., fixed patterns in- 
stead of solid colors), alpha transparency, and, 
when needed, a palette of several colors (see the 
building parameters in line 57). All these param- 
eters are user-definable, and they are how you 
give your map your preferred style. 

From this point of view, the most confusing 
part may be deciding which colors to use for 
each feature. As far as colors are concerned, 
prettymaps follows the same standard ad- 
opted by countless 
other programs, on 
and off the web. Each 
color is described as 
a combination of 
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Figure 4: Finding the color codes for your maps is easy with 
tools such as the W3Schools color picker. 
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Figure 5: For this example, a very different map, showing 
just the physical contours of a geographic object (Lagoa 





dos Patos, Brazil). 
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three fundamental col- 
ors (red, green, and 
blue) and the intensity 
of each of them is rep- 
resented with a two- 
digit hexadecimal number that can go from @@ 
to FF. In this format, the string FF@@@8 Means 
red, @@FF@@ is green, and @@8FF is blue. To see 
the corresponding string for any other color to 
use it when coloring your own maps, you can 
use the W3Schools online color picker [12] 
shown in Figure 4. 

Another very important parameter in pretty- 
maps scripts is the so-called zorder, the order in 
which each layer must be drawn and placed with 
respect to the others. That's why the background 
has the lowest zorder (-1 in line 50): By definition, 
the background must be drawn first and then be 
covered by the others, one at a time. 

The final instructions of Listing 1 specify col- 
ors for the rights and proper credit that OSM de- 
serves (line 60) and the name and format of the 


Listing 2: Finding Places by Their OSM Identifier 
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Figure 6: Lagoa dos Patos shown in Figure 5, as seen in OpenStreetMap with its numeric 


identifier. 





file or files in which the map should be saved 
(lines 63 and 64). 


Other Ways to Find and Map Places 

Figure 5 shows another map that | created by 
just copying, and executing without any change, 
another code sample found on the prettymaps 
homepage. | am including it here because, be- 
sides showing how different these maps can be 
from each other at the purely visual level, it uses 
another OSM feature that is worth knowing. The 
map shows the Lagoa dos Patos in Brazil, and 
Listing 2 shows the relevant parts of the pretty- 
maps script that generates it. 

Figure 6 is what the same lagoon looks like in- 
side OpenStreetMap, plus the unique numeric 
identifier. Inside the OSM database, each “object” 
has a unique numeric identifier that is visible in 
the online map when you select that object. As 
you can see comparing Figure 6 with line 8 of List- 
ing 2, you can use that identifier to directly define 
places in prettymaps! This capability may be very 


fig, ax = plt.subplots(figsize = (10, 12), constrained_ 18 perimeter": {'fil11": False, ‘ec': ‘#6das8cea", ‘lw’: 2, 
layout = True) "Zorder 73), 
14° }, 
def postprocessing (layers): is 
layers['perimeter'] = layers['perimeter'].buffer(2000) il6* osmieerediit = 47a os =. 08. yi 202. “Color tena 73a7 + 
return layers i) 
Te 
layers = plot( 19 # Set bounds 
UR2703093 | 20 xmin, xmax = ax.get_xlim() 
21 ymin, ymax = ax.get_ylim() 
postprocessing = postprocessing, 22 (00, ody xmax-xmin, ymax-ymin 


drawing _kwargs = { 
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23 ax.set_xlim(xmin+.15*dx, xmax-.15*dx) 


24 ax.set_ylim(ymin+.o*dy, ymax-.0O*dy) 


ISSUE 255 


LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM 


useful if you start using prettymaps to generate 
many maps of many places, maybe fetching their 
identifiers with some other script. 

Other relevant parts of Listing 2 are lines 4 
and 13, which process the perimeter of the map in 
a different way than Listing 1, and the final six 
lines, which set the borders of the drawing to 
make sure that no part of the lagoon is cut off the 
map. To see in detail how those instruction work, 
download that example and try to change the nu- 
meric constants in lines 23 and 24. 

As final proof of the great versatility of pretty- 
maps, check out Figure 7, which was generated 
from another code sample found on the web- 
site. Figure 7 shows a map of three adjacent 
neighborhoods of the city of Porto Alegre, Brazil 
(Figure 8), and nothing else. What's more, each 
neighborhood Is painted with different colors. 

The code excerpt in Listing 3 shows how to 
obtain this result. To begin with, you must de- 
fine a hash, an associative array in which each 
element is associated with a key. In Listing 3, 
that hash is called places (lines 4 to 8). Each line 
of that array is about one of the places that 
should be drawn. More specifically, each line de- 
fines a list of colors, in the same format de- 
scribed before, associated with a key that is the 
name of a neighborhood to fetch from Open- 
StreetMap and then draw. Then, you must wrap 
the same calls to the plot and drawing _kwargs 
functions of the previous example in one loop 
(line 10) that handles one neighborhood at a 
time. This happens because every iteration of 
that loop loads one key from the places array, 
calling it place, and loads the corresponding list 
of colors into an auxiliary variable called pal- 
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Figure 7: Drawing adjacent neighborhoods, each with its own style, only takes a little 


extra effort. 


ette. Then, as you can see in lines 12 and 17, 
each call to plot and drawing _kwargs uses the 
place name and colors that are currently loaded 
into those two variables. The autoscale function 
in the last line ensures, with a different method 
than that used by Listing 2, that the whole draw- 
ing will fit in the image. 


Try prettymaps! 

The examples | have presented are enough to 
generate thousands of maps, each with a distinct 
Style, with very little effort once you have installed 
prettymaps and all its dependencies. What's 
more, you may use Matplotlib's capabilities to add 
both captions [13] and generic images [14] to all 
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Figure 7 as they appear in the standard OpenStreetMap view. 
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Listing 3: Mapping Adjacent Places in Different Styles 


O1 fig, ax = plt.subplots(figsize = (15, 12), constrained_layout = True) 


02 fig.patch.set_facecolor('#F9F8F8' ) 
03 


04 places = { 


O05 - 'Farroupilha, Porto Alegre': ['#EEE4E1', '#E7D8C9', '#E6BEAE'], 


O06 - 'Cidade Baixa, Porto Alegre': ['#49392C', 


'#77625C', '#B2B1CF', '#E1F2FE', '#98D2EB'], 


O07 - 'Bom Fim, Porto Alegre': ['#BA2DOB', '#D5F2E3', '#73BA9B', '#F79DS5C'], 


08 } 


O09 


10 for i, (place, palette) in enumerate(places.items()): 


Il plore 
12 place, 
sane 


14 drawing_kwaregs = { 


is ‘perimeter': {'fill': False, 'lw': 0, 'zorder': Oo}, 


16 "Streets (0, fe = “PP lEODO ee. sera ray.« 


liwee eS. sZOrden” a7sh. 


ys "buildine'’: {"*palette': palette, ‘ec’: '#2F3737', ‘lw': 1, ‘zorder’: 4}, 


18 }, 
Als; 
20 ) 
al 


22 ax.autoscale() 


your maps, for any conceivable purpose. It would 
then be quite easy, for example, to create info- 
graphics that show different maps, each associ- 
ated to charts generated from some statistical 
data about the same area. Isn't free software 
wonderful? Sam 





[1] OpenStreetMap: 
https://openstreetmap.org 

(2| Stop at Zona-M blog: 
https://stop.zona-m.net/tag/openstreetmap/ 

[3] prettymaps: https://github.com/ 
marceloprates/prettymaps 

[4] prettymaps examples for Jupyter Notebook: 
https://nbviewer.org/github/marceloprates/ 


prettymaps/blob/main/notebooks/examples. 


ipynb 
[5] More examples on GitHub: 
https://github.com/zaataylor/maps 


[6] OSMnx: https://github.com/gboeing/osmnx 
[7] Shapely: https://pypi.org/project/Shapely/ 
[8] vsketch: https://github.com/abey79/vsketch 
[9] Matplotlib: https://matplotlib.org/ 

[10] IPython: https://ipython.org/ 
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11] prettymaps subreddit: 
www.reddit.com/r/prettymaps_/ 


[12] Online color picker: www.w3schools.com/ 
colors/colors_picker.asp 


'13] Adding captions in Matplotlib: 
https://matplotlib.org/stable/tutorials/text/ 
text_intro.html 


(14| Adding generic images in Matplotlib: 
https://stackoverflow.com/questions/ 
3609585/how-to-insert-a-small-image-on-the- 
corner-of-a-plot-with-matplotlib 
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Date: March 3-6, 2022 


Location: Pasadena, California 


Website: https://register.socallinuxexpo. 
org/reg6/ 


The SCaLE team is pleased to announce 
that the 19th Annual Southern 
California Linux Expo — SCaLE 19X - will 
be held on March 3-6, 2022 at the 
Pasadena Convention Center in 
Pasadena, California, near Los Angeles. 


Registration is open now. 
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